[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:39:05 dist-test-slave-nqgh krb5kdc[25892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:39:05 dist-test-slave-nqgh krb5kdc[25892](info): set up 2 sockets
Sep 28 16:39:05 dist-test-slave-nqgh krb5kdc[25892](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 16:39:06.692504 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:39:06 dist-test-slave-nqgh krb5kdc[25892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:39:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:39:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:39:06.834903 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:41111
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:41111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:06.969167 25905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:06.969569 25905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:06.969633 25905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:06.974502 25905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:06.974676 25905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:06.974716 25905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:06.974747 25905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:06.974781 25905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:39:06.981315 25905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:41111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:41111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:06.982967 25905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:06.984645 25905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:39:06.996521 25905 server_base.cc:1034] running on GCE node
W20260928 16:39:06.996569 25910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:06.996569 25911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:06.996780 25913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:06.997888 25905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:06.999328 25905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:07.000588 25905 hybrid_clock.cc:648] HybridClock initialized: now 1790613547000553 us; error 56 us; skew 500 ppm
Sep 28 16:39:07 dist-test-slave-nqgh krb5kdc[25892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613547, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:07.004659 25905 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:39:07.005995 25905 webserver.cc:460] Webserver started at http://127.25.71.126:45237/ using document root <none> and password file <none>
I20260928 16:39:07.006776 25905 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:07.006867 25905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:07.007099 25905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:07.009783 25905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "6105cf8e0101493e9cb8ed4f1091d13f"
format_stamp: "Formatted at 2026-09-28 16:39:07 on dist-test-slave-nqgh"
server_key: "81b18acf93503bd36a64a4f3ca017727"
server_key_iv: "1e1324896355a4af39ed124e565b6280"
server_key_version: "encryptionkey@0"
I20260928 16:39:07.010491 25905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "6105cf8e0101493e9cb8ed4f1091d13f"
format_stamp: "Formatted at 2026-09-28 16:39:07 on dist-test-slave-nqgh"
server_key: "81b18acf93503bd36a64a4f3ca017727"
server_key_iv: "1e1324896355a4af39ed124e565b6280"
server_key_version: "encryptionkey@0"
I20260928 16:39:07.015993 25905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 16:39:07.020092 25920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:07.022286 25905 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260928 16:39:07.022518 25905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "6105cf8e0101493e9cb8ed4f1091d13f"
format_stamp: "Formatted at 2026-09-28 16:39:07 on dist-test-slave-nqgh"
server_key: "81b18acf93503bd36a64a4f3ca017727"
server_key_iv: "1e1324896355a4af39ed124e565b6280"
server_key_version: "encryptionkey@0"
I20260928 16:39:07.022770 25905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:07.047401 25905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:07.051964 25905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:07.052354 25905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:07.062466 25905 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:41111
I20260928 16:39:07.062569 25972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:41111 every 8 connection(s)
I20260928 16:39:07.063956 25905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:39:07.064843 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 25905
I20260928 16:39:07.065013 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:39:07.066555 25885 external_mini_cluster.cc:1428] Setting key ab9ba0e5b97a11f9404e8ed9e02b5d0d
I20260928 16:39:07.067914 25973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:07.076849 25973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f: Bootstrap starting.
I20260928 16:39:07.080531 25973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:07.081662 25973 log.cc:824] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:07.084574 25973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f: No bootstrap required, opened a new log
Sep 28 16:39:07 dist-test-slave-nqgh krb5kdc[25892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:07.090055 25973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6105cf8e0101493e9cb8ed4f1091d13f" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41111 } }
I20260928 16:39:07.090447 25973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:07.090534 25973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6105cf8e0101493e9cb8ed4f1091d13f, State: Initialized, Role: FOLLOWER
I20260928 16:39:07.091239 25973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6105cf8e0101493e9cb8ed4f1091d13f" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41111 } }
I20260928 16:39:07.091450 25973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:39:07.091547 25973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:39:07.091676 25973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:07.093118 25973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6105cf8e0101493e9cb8ed4f1091d13f" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41111 } }
I20260928 16:39:07.093688 25973 leader_election.cc:302] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6105cf8e0101493e9cb8ed4f1091d13f; no voters:
I20260928 16:39:07.094197 25973 leader_election.cc:288] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:39:07.094396 25978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:07.094866 25978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f [term 1 LEADER]: Becoming Leader. State: Replica: 6105cf8e0101493e9cb8ed4f1091d13f, State: Running, Role: LEADER
I20260928 16:39:07.095383 25978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6105cf8e0101493e9cb8ed4f1091d13f" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41111 } }
I20260928 16:39:07.096133 25973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:07.097195 25979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6105cf8e0101493e9cb8ed4f1091d13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6105cf8e0101493e9cb8ed4f1091d13f" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41111 } } }
I20260928 16:39:07.097368 25979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:07.098342 25986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:39:07.098763 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:07.070837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.25.71.126:41111)
0928 16:39:07.071493 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:07.071511 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:07.072861 (+ 1350us) server_negotiation.cc:407] Connection header received
0928 16:39:07.076572 (+ 3711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:07.076592 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:07.077005 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:07.077454 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:07.080265 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:07.081476 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:07.083424 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:07.083983 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:07.084170 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:07.089881 (+ 5711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:07.089888 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:07.089907 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:07.089960 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:07.092591 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:07.093710 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:07.093719 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:07.093724 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:07.093941 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:07.094403 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:07.094407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:07.094408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:07.095155 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:07.095405 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:07.096578 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0928 16:39:07.096988 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":202,"threads_started":1}
I20260928 16:39:07.097855 25980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6105cf8e0101493e9cb8ed4f1091d13f. Latest consensus state: current_term: 1 leader_uuid: "6105cf8e0101493e9cb8ed4f1091d13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6105cf8e0101493e9cb8ed4f1091d13f" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41111 } } }
I20260928 16:39:07.102591 25980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:07.102845 25986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:39:07.111315 25986 catalog_manager.cc:1347] Generated new cluster ID: 3765afa54320451b83b97b2a6ceded5b
I20260928 16:39:07.111442 25986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:39:07.136292 25986 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:39:07.137388 25986 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:39:07.146993 25986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6105cf8e0101493e9cb8ed4f1091d13f: Generated new TSK 0
I20260928 16:39:07.148175 25986 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
I20260928 16:39:07.192785 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41111
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:07.320254 25999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:07.320549 25999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:07.320611 25999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:07.324896 25999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:07.324978 25999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:07.325065 25999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:39:07.330804 25999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41111
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:07.332003 25999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:07.333092 25999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:07.344026 26007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:07.344033 26004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:07.344141 25999 server_base.cc:1034] running on GCE node
W20260928 16:39:07.344028 26005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:07.345034 25999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:07.346031 25999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:07.347251 25999 hybrid_clock.cc:648] HybridClock initialized: now 1790613547347212 us; error 60 us; skew 500 ppm
Sep 28 16:39:07 dist-test-slave-nqgh krb5kdc[25892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613547, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:07.351022 25999 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:39:07.352177 25999 webserver.cc:460] Webserver started at http://127.25.71.65:37345/ using document root <none> and password file <none>
I20260928 16:39:07.352974 25999 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:07.353078 25999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:07.353350 25999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:07.355698 25999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "bf0fd263cf414481904d845b18cc285d"
format_stamp: "Formatted at 2026-09-28 16:39:07 on dist-test-slave-nqgh"
server_key: "fc2ac1f76cdc9d9c1de679ca8fbddb9b"
server_key_iv: "2b23c4c679ad3c7d8af0925943c2148a"
server_key_version: "encryptionkey@0"
I20260928 16:39:07.356261 25999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "bf0fd263cf414481904d845b18cc285d"
format_stamp: "Formatted at 2026-09-28 16:39:07 on dist-test-slave-nqgh"
server_key: "fc2ac1f76cdc9d9c1de679ca8fbddb9b"
server_key_iv: "2b23c4c679ad3c7d8af0925943c2148a"
server_key_version: "encryptionkey@0"
I20260928 16:39:07.360661 25999 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:07.363830 26014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:07.365811 25999 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:07.366005 25999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "bf0fd263cf414481904d845b18cc285d"
format_stamp: "Formatted at 2026-09-28 16:39:07 on dist-test-slave-nqgh"
server_key: "fc2ac1f76cdc9d9c1de679ca8fbddb9b"
server_key_iv: "2b23c4c679ad3c7d8af0925943c2148a"
server_key_version: "encryptionkey@0"
I20260928 16:39:07.366142 25999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:07.381870 25999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:07.385145 25999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:07.385375 25999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:07.386016 25999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:07.387001 25999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:07.387055 25999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:07.387110 25999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:07.387136 25999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:07.398221 25999 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:36139
I20260928 16:39:07.398257 26127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:36139 every 8 connection(s)
I20260928 16:39:07.399369 25999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:39:07.401491 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 25999
I20260928 16:39:07.401595 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:39:07.401875 25885 external_mini_cluster.cc:1428] Setting key d600ebdd46f6b7b637cc53e0a597f1b1
Sep 28 16:39:07 dist-test-slave-nqgh krb5kdc[25892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613547, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.66@KRBTEST.COM" created.
I20260928 16:39:07.417719 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:07.402023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:38787 (local address 127.25.71.126:41111)
0928 16:39:07.402233 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:07.402239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:07.403642 (+ 1403us) server_negotiation.cc:407] Connection header received
0928 16:39:07.404975 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:07.404979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:07.405037 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:07.405145 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:07.406786 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:07.407361 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:07.409009 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:07.409446 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:07.409623 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:07.412896 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:07.412901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:07.412904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:07.412952 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:07.414779 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:07.415556 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:07.415561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:07.415563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:07.415697 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:07.416121 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:07.416126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:07.416129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:07.416464 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:07.416585 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:07.417371 (+ 786us) server_negotiation.cc:299] Negotiation successful
0928 16:39:07.417547 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 16:39:07.418905 26130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:07.402489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:41111 (local address 127.25.71.65:38787)
0928 16:39:07.403460 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:07.403520 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:07.404653 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:07.405392 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:07.405403 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:07.405967 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:07.406570 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:07.406587 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:07.407506 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:07.407510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:07.408709 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:07.408724 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:07.409689 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:07.409693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:07.409945 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:07.410602 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:07.410621 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:07.412610 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:07.414969 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:07.414976 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:07.414992 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:07.415369 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:07.415802 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:07.415805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:07.415806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:07.415969 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:07.416635 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:07.416644 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:07.417033 (+ 389us) client_negotiation.cc:769] Sending connection context
0928 16:39:07.417313 (+ 280us) client_negotiation.cc:240] Negotiation successful
0928 16:39:07.417746 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":205,"threads_started":1}
I20260928 16:39:07.420809 26128 heartbeater.cc:347] Connected to a master server at 127.25.71.126:41111
I20260928 16:39:07.421094 26128 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:07.421713 26128 heartbeater.cc:511] Master 127.25.71.126:41111 requested a full tablet report, sending...
I20260928 16:39:07.423858 25933 ts_manager.cc:194] Registered new tserver with Master: bf0fd263cf414481904d845b18cc285d (127.25.71.65:36139)
I20260928 16:39:07.425204 25933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:38787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
I20260928 16:39:07.438231 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41111
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:07.572877 26133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:07.573276 26133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:07.573351 26133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:07.578060 26133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:07.578173 26133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:07.578296 26133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:39:07.584879 26133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41111
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:07.586452 26133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:07.587936 26133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:07.600677 26139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:07.600668 26138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:07.600662 26141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:07.600719 26133 server_base.cc:1034] running on GCE node
I20260928 16:39:07.601637 26133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:07.602501 26133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:07.603725 26133 hybrid_clock.cc:648] HybridClock initialized: now 1790613547603636 us; error 120 us; skew 500 ppm
Sep 28 16:39:07 dist-test-slave-nqgh krb5kdc[25892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613547, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:07.607589 26133 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:39:07.608943 26133 webserver.cc:460] Webserver started at http://127.25.71.66:38669/ using document root <none> and password file <none>
I20260928 16:39:07.609715 26133 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:07.609804 26133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:07.610038 26133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:07.612969 26133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "ef45beb2f45242a5abdee1e86ff8577a"
format_stamp: "Formatted at 2026-09-28 16:39:07 on dist-test-slave-nqgh"
server_key: "bd86a5d89d6f3799e31ff11fc7aecef5"
server_key_iv: "512d74cfbad25f920a039a1957ad047d"
server_key_version: "encryptionkey@0"
I20260928 16:39:07.613669 26133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "ef45beb2f45242a5abdee1e86ff8577a"
format_stamp: "Formatted at 2026-09-28 16:39:07 on dist-test-slave-nqgh"
server_key: "bd86a5d89d6f3799e31ff11fc7aecef5"
server_key_iv: "512d74cfbad25f920a039a1957ad047d"
server_key_version: "encryptionkey@0"
I20260928 16:39:07.618921 26133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 16:39:07.622673 26148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:07.624629 26133 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:07.624836 26133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "ef45beb2f45242a5abdee1e86ff8577a"
format_stamp: "Formatted at 2026-09-28 16:39:07 on dist-test-slave-nqgh"
server_key: "bd86a5d89d6f3799e31ff11fc7aecef5"
server_key_iv: "512d74cfbad25f920a039a1957ad047d"
server_key_version: "encryptionkey@0"
I20260928 16:39:07.625008 26133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:07.640689 26133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:07.645598 26133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:07.645897 26133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:07.647010 26133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:07.648198 26133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:07.648268 26133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:07.648331 26133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:07.648365 26133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:07.660539 26133 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:39453
I20260928 16:39:07.660588 26261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:39453 every 8 connection(s)
I20260928 16:39:07.661784 26133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:39:07.667237 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 26133
I20260928 16:39:07.667357 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:39:07.667610 25885 external_mini_cluster.cc:1428] Setting key 97ac8ff2b7451db3c935db35ed84e4df
Sep 28 16:39:07 dist-test-slave-nqgh krb5kdc[25892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613547, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.71.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.67@KRBTEST.COM" created.
I20260928 16:39:07.681149 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:07.664346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:58261 (local address 127.25.71.126:41111)
0928 16:39:07.664565 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:07.664569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:07.666068 (+ 1499us) server_negotiation.cc:407] Connection header received
0928 16:39:07.667268 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:07.667271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:07.667369 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:07.667514 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:07.669443 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:07.670232 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:07.672227 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:07.672510 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:07.672635 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:07.676342 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:07.676346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:07.676349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:07.676385 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:07.678134 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:07.678974 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:07.678978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:07.678980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:07.679077 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:07.679558 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:07.679561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:07.679562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:07.679827 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:07.679922 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:07.680757 (+ 835us) server_negotiation.cc:299] Negotiation successful
0928 16:39:07.680937 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 16:39:07.682363 26264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:07.664909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:41111 (local address 127.25.71.66:58261)
0928 16:39:07.665849 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:07.665922 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:07.666913 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:07.667856 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:07.667867 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:07.668475 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:07.669233 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:07.669261 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:07.670448 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:07.670452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:07.671984 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:07.672014 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:07.672726 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:07.672731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:07.672968 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:07.673812 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:07.673834 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:07.676005 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:07.678313 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:07.678324 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:07.678349 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:07.678847 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:07.679248 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:07.679251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:07.679253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:07.679444 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:07.679942 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:07.679952 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:07.680429 (+ 477us) client_negotiation.cc:769] Sending connection context
0928 16:39:07.680699 (+ 270us) client_negotiation.cc:240] Negotiation successful
0928 16:39:07.681183 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":168,"threads_started":1}
I20260928 16:39:07.684235 26262 heartbeater.cc:347] Connected to a master server at 127.25.71.126:41111
I20260928 16:39:07.684640 26262 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:07.685497 26262 heartbeater.cc:511] Master 127.25.71.126:41111 requested a full tablet report, sending...
I20260928 16:39:07.687145 25933 ts_manager.cc:194] Registered new tserver with Master: ef45beb2f45242a5abdee1e86ff8577a (127.25.71.66:39453)
I20260928 16:39:07.688040 25933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:58261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
I20260928 16:39:07.699054 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41111
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:07.824059 26267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:07.824370 26267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:07.824417 26267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:07.828761 26267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:07.828846 26267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:07.828918 26267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:39:07.834795 26267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41111
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:07.836067 26267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:07.837172 26267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:07.849221 26273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:07.849176 26275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:07.849289 26272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:07.849535 26267 server_base.cc:1034] running on GCE node
I20260928 16:39:07.850157 26267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:07.851119 26267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:07.852348 26267 hybrid_clock.cc:648] HybridClock initialized: now 1790613547852308 us; error 53 us; skew 500 ppm
Sep 28 16:39:07 dist-test-slave-nqgh krb5kdc[25892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613547, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:07.856278 26267 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:39:07.857569 26267 webserver.cc:460] Webserver started at http://127.25.71.67:40347/ using document root <none> and password file <none>
I20260928 16:39:07.858301 26267 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:07.858387 26267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:07.858686 26267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:07.861528 26267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "34356786644a4132a36d29b712f49096"
format_stamp: "Formatted at 2026-09-28 16:39:07 on dist-test-slave-nqgh"
server_key: "6c7f1d67b5364bf4a121e81ff4edd545"
server_key_iv: "f689e29d224e7c88e924ad5f226eb583"
server_key_version: "encryptionkey@0"
I20260928 16:39:07.862262 26267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "34356786644a4132a36d29b712f49096"
format_stamp: "Formatted at 2026-09-28 16:39:07 on dist-test-slave-nqgh"
server_key: "6c7f1d67b5364bf4a121e81ff4edd545"
server_key_iv: "f689e29d224e7c88e924ad5f226eb583"
server_key_version: "encryptionkey@0"
I20260928 16:39:07.867612 26267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 16:39:07.871515 26282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:07.873420 26267 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:07.873622 26267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "34356786644a4132a36d29b712f49096"
format_stamp: "Formatted at 2026-09-28 16:39:07 on dist-test-slave-nqgh"
server_key: "6c7f1d67b5364bf4a121e81ff4edd545"
server_key_iv: "f689e29d224e7c88e924ad5f226eb583"
server_key_version: "encryptionkey@0"
I20260928 16:39:07.873787 26267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:07.926878 26267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:07.931879 26267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:07.932161 26267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:07.933050 26267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:07.934240 26267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:07.934301 26267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:07.934372 26267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:07.934404 26267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:07.947218 26267 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:44917
I20260928 16:39:07.947257 26395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:44917 every 8 connection(s)
I20260928 16:39:07.948479 26267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:39:07.958082 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 26267
I20260928 16:39:07.958243 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:39:07.958508 25885 external_mini_cluster.cc:1428] Setting key 4655374d9f1c61de8b0bc235dec7ff6f
Sep 28 16:39:07 dist-test-slave-nqgh krb5kdc[25892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613547, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:07.965982 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:07.950824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:36599 (local address 127.25.71.126:41111)
0928 16:39:07.950997 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:07.951004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:07.952462 (+ 1458us) server_negotiation.cc:407] Connection header received
0928 16:39:07.953561 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:07.953565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:07.953631 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:07.953729 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:07.955482 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:07.956114 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:07.957714 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:07.958056 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:07.958195 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:07.961759 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:07.961763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:07.961766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:07.961802 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:07.963496 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:07.964295 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:07.964298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:07.964299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:07.964399 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:07.964740 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:07.964742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:07.964744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:07.964960 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:07.965036 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:07.965669 (+ 633us) server_negotiation.cc:299] Negotiation successful
0928 16:39:07.965818 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:39:07.966982 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:07.951126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:41111 (local address 127.25.71.67:36599)
0928 16:39:07.952205 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:07.952271 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:07.953289 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:07.953963 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:07.953971 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:07.954536 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:07.955281 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:07.955300 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:07.956325 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:07.956329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:07.957504 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:07.957517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:07.958302 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:07.958307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:07.958520 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:07.959322 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:07.959346 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:07.961551 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:07.963662 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:07.963669 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:07.963685 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:07.964129 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:07.964493 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:07.964495 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:07.964496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:07.964651 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:07.965056 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:07.965062 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:07.965369 (+ 307us) client_negotiation.cc:769] Sending connection context
0928 16:39:07.965540 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 16:39:07.965857 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":159,"threads_started":1}
I20260928 16:39:07.968716 26396 heartbeater.cc:347] Connected to a master server at 127.25.71.126:41111
I20260928 16:39:07.969095 26396 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:07.969687 26396 heartbeater.cc:511] Master 127.25.71.126:41111 requested a full tablet report, sending...
I20260928 16:39:07.971168 25933 ts_manager.cc:194] Registered new tserver with Master: 34356786644a4132a36d29b712f49096 (127.25.71.67:44917)
I20260928 16:39:07.971813 25933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:36599
I20260928 16:39:07.974288 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:39:07 dist-test-slave-nqgh krb5kdc[25892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613547, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:39:07 dist-test-slave-nqgh krb5kdc[25892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613547, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.71.65@KRBTEST.COM
I20260928 16:39:08.005605 26403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:07.993883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37670 (local address 127.25.71.65:36139)
0928 16:39:07.994350 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:07.994361 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:07.994393 (+ 32us) server_negotiation.cc:407] Connection header received
0928 16:39:07.994462 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:07.994469 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:07.994790 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:07.994955 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:07.995795 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:07.996787 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:07.998018 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:07.998482 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:07.998587 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:08.001427 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:08.001440 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:08.001460 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:08.001495 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:08.003382 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:08.003975 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:08.003984 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:08.003985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:08.004101 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:08.004381 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:08.004385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:08.004386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:08.004734 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:08.005001 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:08.005188 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 16:39:08.005384 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":126,"threads_started":1}
W20260928 16:39:08.007045 26042 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37670
Sep 28 16:39:08 dist-test-slave-nqgh krb5kdc[25892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:39:08 dist-test-slave-nqgh krb5kdc[25892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.65@KRBTEST.COM
I20260928 16:39:08.039923 26403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:08.028615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37686 (local address 127.25.71.65:36139)
0928 16:39:08.028832 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:08.028836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:08.028951 (+ 115us) server_negotiation.cc:407] Connection header received
0928 16:39:08.029115 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:08.029118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:08.029178 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:08.029254 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:08.030251 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:08.030935 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:08.032132 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:08.032411 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:08.032522 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:08.035887 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:08.035892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:08.035895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:08.035942 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:08.037754 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:08.038338 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:08.038341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:08.038343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:08.038445 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:08.038878 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:08.038881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:08.038883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:08.039174 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:08.039311 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:08.039613 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 16:39:08.039753 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 16:39:08.041458 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 25999
I20260928 16:39:08.047420 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 26133
I20260928 16:39:08.053185 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 26267
I20260928 16:39:08.058876 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 25905
2026-09-28T16:39:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2380 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:39:08 dist-test-slave-nqgh krb5kdc[26410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:39:08 dist-test-slave-nqgh krb5kdc[26410](info): set up 2 sockets
Sep 28 16:39:08 dist-test-slave-nqgh krb5kdc[26410](info): commencing operation
krb5kdc: starting...
W20260928 16:39:09.044178 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:39:09 dist-test-slave-nqgh krb5kdc[26410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:39:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:39:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:39:09.187879 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:45165
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:40721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:45165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:09.325374 26423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:09.325769 26423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:09.325835 26423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:09.330675 26423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:09.330797 26423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:09.330825 26423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:09.330847 26423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:09.330883 26423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:39:09.337535 26423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:40721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:45165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:45165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:09.339176 26423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:09.340574 26423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:09.351302 26428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:09.351387 26431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:09.351403 26429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:09.351729 26423 server_base.cc:1034] running on GCE node
I20260928 16:39:09.352530 26423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:09.354096 26423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:09.355384 26423 hybrid_clock.cc:648] HybridClock initialized: now 1790613549355350 us; error 58 us; skew 500 ppm
Sep 28 16:39:09 dist-test-slave-nqgh krb5kdc[26410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613549, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:09.359601 26423 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:39:09.361104 26423 webserver.cc:460] Webserver started at http://127.25.71.126:37897/ using document root <none> and password file <none>
I20260928 16:39:09.361891 26423 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:09.361972 26423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:09.362242 26423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:09.365127 26423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "a23754e1a04d45788df31b0f39f328d1"
format_stamp: "Formatted at 2026-09-28 16:39:09 on dist-test-slave-nqgh"
server_key: "59140dccd5a030a32f125b29a6e9ef06"
server_key_iv: "144a93fcd2c5c098802ac9c45e499983"
server_key_version: "encryptionkey@0"
I20260928 16:39:09.365931 26423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "a23754e1a04d45788df31b0f39f328d1"
format_stamp: "Formatted at 2026-09-28 16:39:09 on dist-test-slave-nqgh"
server_key: "59140dccd5a030a32f125b29a6e9ef06"
server_key_iv: "144a93fcd2c5c098802ac9c45e499983"
server_key_version: "encryptionkey@0"
I20260928 16:39:09.371528 26423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 16:39:09.375665 26438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:09.377826 26423 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:09.378058 26423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "a23754e1a04d45788df31b0f39f328d1"
format_stamp: "Formatted at 2026-09-28 16:39:09 on dist-test-slave-nqgh"
server_key: "59140dccd5a030a32f125b29a6e9ef06"
server_key_iv: "144a93fcd2c5c098802ac9c45e499983"
server_key_version: "encryptionkey@0"
I20260928 16:39:09.378285 26423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:09.392975 26423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:09.397688 26423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:09.397938 26423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:09.407454 26423 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:45165
I20260928 16:39:09.407531 26490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:45165 every 8 connection(s)
I20260928 16:39:09.408783 26423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:39:09.412302 26491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:09.417207 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 26423
I20260928 16:39:09.417352 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:39:09.417687 25885 external_mini_cluster.cc:1428] Setting key 733e27e6ff8a1a89053871038cc3c52c
I20260928 16:39:09.418895 26491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1: Bootstrap starting.
I20260928 16:39:09.421571 26491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:09.422596 26491 log.cc:824] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:09.425603 26491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1: No bootstrap required, opened a new log
Sep 28 16:39:09 dist-test-slave-nqgh krb5kdc[26410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:09.430438 26491 raft_consensus.cc:348] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a23754e1a04d45788df31b0f39f328d1" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 45165 } }
I20260928 16:39:09.430785 26491 raft_consensus.cc:374] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:09.430851 26491 raft_consensus.cc:729] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a23754e1a04d45788df31b0f39f328d1, State: Initialized, Role: FOLLOWER
I20260928 16:39:09.431427 26491 consensus_queue.cc:260] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a23754e1a04d45788df31b0f39f328d1" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 45165 } }
I20260928 16:39:09.431594 26491 raft_consensus.cc:388] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:39:09.431668 26491 raft_consensus.cc:482] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:39:09.431771 26491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:09.432809 26491 raft_consensus.cc:504] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a23754e1a04d45788df31b0f39f328d1" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 45165 } }
I20260928 16:39:09.433205 26491 leader_election.cc:302] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a23754e1a04d45788df31b0f39f328d1; no voters:
I20260928 16:39:09.433568 26491 leader_election.cc:288] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:39:09.433671 26496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:09.433871 26496 raft_consensus.cc:686] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1 [term 1 LEADER]: Becoming Leader. State: Replica: a23754e1a04d45788df31b0f39f328d1, State: Running, Role: LEADER
I20260928 16:39:09.434211 26496 consensus_queue.cc:237] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a23754e1a04d45788df31b0f39f328d1" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 45165 } }
I20260928 16:39:09.435177 26491 sys_catalog.cc:564] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:09.436247 26497 sys_catalog.cc:455] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a23754e1a04d45788df31b0f39f328d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a23754e1a04d45788df31b0f39f328d1" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 45165 } } }
I20260928 16:39:09.436450 26497 sys_catalog.cc:458] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:09.436987 26498 sys_catalog.cc:455] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a23754e1a04d45788df31b0f39f328d1. Latest consensus state: current_term: 1 leader_uuid: "a23754e1a04d45788df31b0f39f328d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a23754e1a04d45788df31b0f39f328d1" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 45165 } } }
I20260928 16:39:09.438226 26498 sys_catalog.cc:458] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:09.437966 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:09.419549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57350 (local address 127.25.71.126:45165)
0928 16:39:09.420114 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:09.420134 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:09.420182 (+ 48us) server_negotiation.cc:407] Connection header received
0928 16:39:09.421046 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:09.421069 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:09.421534 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:09.421979 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:09.423109 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:09.424493 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:09.425924 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:09.426673 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:09.426892 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:09.430444 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:09.430451 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:09.430477 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:09.430528 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:09.432857 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:09.433536 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:09.433544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:09.433551 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:09.433744 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:09.434225 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:09.434228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:09.434230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:09.435108 (+ 878us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:09.435318 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:09.435922 (+ 604us) server_negotiation.cc:299] Negotiation successful
0928 16:39:09.436511 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":164,"threads_started":1}
W20260928 16:39:09.441376 26511 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:39:09.441661 26511 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:39:09.441792 26505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:39:09.442984 26505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:39:09.451159 26505 catalog_manager.cc:1347] Generated new cluster ID: e02c841987f04dfcbe54afc0e65cd69f
I20260928 16:39:09.451288 26505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:39:09.460273 26505 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:39:09.461858 26505 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:39:09.469796 26505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a23754e1a04d45788df31b0f39f328d1: Generated new TSK 0
I20260928 16:39:09.470871 26505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
I20260928 16:39:09.506937 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:45165
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:40721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:09.656921 26517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:09.657310 26517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:09.657361 26517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:09.661954 26517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:09.662073 26517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:09.662144 26517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:39:09.668560 26517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:40721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:45165
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:09.670482 26517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:09.672082 26517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:09.685058 26522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:09.685187 26525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:09.685343 26523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:09.685791 26517 server_base.cc:1034] running on GCE node
I20260928 16:39:09.686336 26517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:09.687166 26517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:09.688397 26517 hybrid_clock.cc:648] HybridClock initialized: now 1790613549688358 us; error 57 us; skew 500 ppm
Sep 28 16:39:09 dist-test-slave-nqgh krb5kdc[26410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613549, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:09.692001 26517 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:39:09.693387 26517 webserver.cc:460] Webserver started at http://127.25.71.65:44483/ using document root <none> and password file <none>
I20260928 16:39:09.694105 26517 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:09.694180 26517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:09.694408 26517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:09.696890 26517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "425cfc8675214eedb1ed4aaece127294"
format_stamp: "Formatted at 2026-09-28 16:39:09 on dist-test-slave-nqgh"
server_key: "912b7018edcc3012aef69e58f384cc3f"
server_key_iv: "2ac4822e2cac04cae1f8cfdc4a3b86aa"
server_key_version: "encryptionkey@0"
I20260928 16:39:09.697738 26517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "425cfc8675214eedb1ed4aaece127294"
format_stamp: "Formatted at 2026-09-28 16:39:09 on dist-test-slave-nqgh"
server_key: "912b7018edcc3012aef69e58f384cc3f"
server_key_iv: "2ac4822e2cac04cae1f8cfdc4a3b86aa"
server_key_version: "encryptionkey@0"
I20260928 16:39:09.703364 26517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 16:39:09.706751 26532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:09.708657 26517 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:09.708856 26517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "425cfc8675214eedb1ed4aaece127294"
format_stamp: "Formatted at 2026-09-28 16:39:09 on dist-test-slave-nqgh"
server_key: "912b7018edcc3012aef69e58f384cc3f"
server_key_iv: "2ac4822e2cac04cae1f8cfdc4a3b86aa"
server_key_version: "encryptionkey@0"
I20260928 16:39:09.708993 26517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:09.743458 26517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:09.747540 26517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:09.747759 26517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:09.748417 26517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:09.749393 26517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:09.749447 26517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:09.749492 26517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:09.749506 26517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:09.759567 26517 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:44253
I20260928 16:39:09.759640 26645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:44253 every 8 connection(s)
I20260928 16:39:09.760607 26517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:39:09.766517 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 26517
I20260928 16:39:09.766692 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:39:09.767010 25885 external_mini_cluster.cc:1428] Setting key bb015a32c7e61a3884dcb472d9aee615
Sep 28 16:39:09 dist-test-slave-nqgh krb5kdc[26410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613549, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.71.66@KRBTEST.COM; defaulting to no policy
I20260928 16:39:09.780791 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:09.763231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:34615 (local address 127.25.71.126:45165)
0928 16:39:09.763477 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:09.763481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:09.764637 (+ 1156us) server_negotiation.cc:407] Connection header received
0928 16:39:09.765754 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:09.765757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:09.765821 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:09.765900 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:09.767572 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:09.768307 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:09.770009 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:09.770425 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:09.770562 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:09.774836 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:09.774841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:09.774844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:09.774895 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:09.777718 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:09.778588 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:09.778593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:09.778595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:09.778745 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:09.779187 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:09.779191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:09.779193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:09.779450 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:09.779555 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:09.780422 (+ 867us) server_negotiation.cc:299] Negotiation successful
0928 16:39:09.780588 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.66@KRBTEST.COM" created.
I20260928 16:39:09.781885 26648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:09.763729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:45165 (local address 127.25.71.65:34615)
0928 16:39:09.764430 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:09.764491 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:09.765516 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:09.766112 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:09.766120 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:09.766667 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:09.767405 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:09.767427 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:09.768465 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:09.768468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:09.769816 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:09.769830 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:09.770717 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:09.770724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:09.771036 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:09.771925 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:09.771951 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:09.774446 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:09.777903 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:09.777914 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:09.777940 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:09.778424 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:09.778835 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:09.778838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:09.778857 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:09.779086 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:09.779666 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:09.779675 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:09.780088 (+ 413us) client_negotiation.cc:769] Sending connection context
0928 16:39:09.780316 (+ 228us) client_negotiation.cc:240] Negotiation successful
0928 16:39:09.780694 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":179,"threads_started":1}
I20260928 16:39:09.784054 26646 heartbeater.cc:347] Connected to a master server at 127.25.71.126:45165
I20260928 16:39:09.784397 26646 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:09.785048 26646 heartbeater.cc:511] Master 127.25.71.126:45165 requested a full tablet report, sending...
I20260928 16:39:09.787312 26455 ts_manager.cc:194] Registered new tserver with Master: 425cfc8675214eedb1ed4aaece127294 (127.25.71.65:44253)
I20260928 16:39:09.789055 26455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:34615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
I20260928 16:39:09.805977 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:45165
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:40721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:09.928488 26651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:09.928793 26651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:09.928843 26651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:09.933089 26651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:09.933174 26651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:09.933238 26651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:39:09.939234 26651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:40721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:45165
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:09.940392 26651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:09.941432 26651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:09.952702 26656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:09.952704 26657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:09.952685 26659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:09.952939 26651 server_base.cc:1034] running on GCE node
I20260928 16:39:09.953745 26651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:09.954478 26651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:09.955681 26651 hybrid_clock.cc:648] HybridClock initialized: now 1790613549955646 us; error 53 us; skew 500 ppm
Sep 28 16:39:09 dist-test-slave-nqgh krb5kdc[26410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613549, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:09.959106 26651 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:39:09.960211 26651 webserver.cc:460] Webserver started at http://127.25.71.66:38581/ using document root <none> and password file <none>
I20260928 16:39:09.960827 26651 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:09.960899 26651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:09.961107 26651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:09.963511 26651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "41ef3c8066f5425f86de9300982b3933"
format_stamp: "Formatted at 2026-09-28 16:39:09 on dist-test-slave-nqgh"
server_key: "002ba14fc044defbf1730a41993aece3"
server_key_iv: "712a66a4141ed3ef8da14d32c169c596"
server_key_version: "encryptionkey@0"
I20260928 16:39:09.964169 26651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "41ef3c8066f5425f86de9300982b3933"
format_stamp: "Formatted at 2026-09-28 16:39:09 on dist-test-slave-nqgh"
server_key: "002ba14fc044defbf1730a41993aece3"
server_key_iv: "712a66a4141ed3ef8da14d32c169c596"
server_key_version: "encryptionkey@0"
I20260928 16:39:09.968614 26651 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:09.971956 26666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:09.973830 26651 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:09.973994 26651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "41ef3c8066f5425f86de9300982b3933"
format_stamp: "Formatted at 2026-09-28 16:39:09 on dist-test-slave-nqgh"
server_key: "002ba14fc044defbf1730a41993aece3"
server_key_iv: "712a66a4141ed3ef8da14d32c169c596"
server_key_version: "encryptionkey@0"
I20260928 16:39:09.974120 26651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:10.003338 26651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:10.006773 26651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:10.006979 26651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:10.007586 26651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:10.008459 26651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:10.008512 26651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:10.008566 26651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:10.008596 26651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:10.019101 26651 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:39371
I20260928 16:39:10.019188 26779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:39371 every 8 connection(s)
I20260928 16:39:10.020143 26651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:39:10.024634 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 26651
I20260928 16:39:10.024744 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:39:10.025008 25885 external_mini_cluster.cc:1428] Setting key 2a018b65ea6ef4d1db59206bb310c6c9
Sep 28 16:39:10 dist-test-slave-nqgh krb5kdc[26410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613549, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.71.67@KRBTEST.COM; defaulting to no policy
I20260928 16:39:10.037873 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:10.022784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:34779 (local address 127.25.71.126:45165)
0928 16:39:10.022957 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:10.022961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:10.023997 (+ 1036us) server_negotiation.cc:407] Connection header received
0928 16:39:10.025296 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:10.025301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:10.025372 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:10.025456 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:10.027248 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.027829 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.029649 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.029989 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.030125 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:10.033716 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:10.033719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:10.033721 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:10.033753 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:10.035285 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:10.036069 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:10.036072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:10.036073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:10.036153 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:10.036505 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:10.036508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:10.036509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:10.036856 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:10.036948 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:10.037532 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 16:39:10.037689 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:39:10.038870 26782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:10.023123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:45165 (local address 127.25.71.66:34779)
0928 16:39:10.023776 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:10.023841 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:10.025014 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:10.025695 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:10.025703 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:10.026327 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:10.027076 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:10.027096 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.027965 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.027968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:10.029437 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:10.029451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.030222 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.030226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:10.030462 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:10.031251 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:10.031275 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:10.033466 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:10.035442 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:10.035450 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:10.035472 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:10.035950 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:10.036239 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:10.036242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:10.036245 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:10.036404 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:10.036959 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:10.036969 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:10.037272 (+ 303us) client_negotiation.cc:769] Sending connection context
0928 16:39:10.037453 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 16:39:10.037855 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":177,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.67@KRBTEST.COM" created.
I20260928 16:39:10.040452 26780 heartbeater.cc:347] Connected to a master server at 127.25.71.126:45165
I20260928 16:39:10.040902 26780 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:10.041813 26780 heartbeater.cc:511] Master 127.25.71.126:45165 requested a full tablet report, sending...
I20260928 16:39:10.043761 26455 ts_manager.cc:194] Registered new tserver with Master: 41ef3c8066f5425f86de9300982b3933 (127.25.71.66:39371)
I20260928 16:39:10.044541 26455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:34779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
I20260928 16:39:10.059047 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:45165
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:40721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:10.201551 26785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:10.201972 26785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:10.202044 26785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:10.206928 26785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:10.207034 26785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:10.207131 26785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:39:10.213959 26785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:40721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:45165
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:10.215600 26785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:10.217041 26785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:10.229868 26790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:10.229938 26793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:10.229899 26791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:10.230335 26785 server_base.cc:1034] running on GCE node
I20260928 16:39:10.231070 26785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:10.232005 26785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:10.233210 26785 hybrid_clock.cc:648] HybridClock initialized: now 1790613550233184 us; error 51 us; skew 500 ppm
Sep 28 16:39:10 dist-test-slave-nqgh krb5kdc[26410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613550, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:10.236841 26785 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:39:10.238186 26785 webserver.cc:460] Webserver started at http://127.25.71.67:40571/ using document root <none> and password file <none>
I20260928 16:39:10.239005 26785 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:10.239094 26785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:10.239364 26785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:10.242144 26785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "136c5537dc544e6e8b12cf0beb5b45da"
format_stamp: "Formatted at 2026-09-28 16:39:10 on dist-test-slave-nqgh"
server_key: "db6ef58572275e6d7843e8d70b3c1337"
server_key_iv: "aa55495ad9e1c158e8c7ceff479d8cb0"
server_key_version: "encryptionkey@0"
I20260928 16:39:10.242971 26785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "136c5537dc544e6e8b12cf0beb5b45da"
format_stamp: "Formatted at 2026-09-28 16:39:10 on dist-test-slave-nqgh"
server_key: "db6ef58572275e6d7843e8d70b3c1337"
server_key_iv: "aa55495ad9e1c158e8c7ceff479d8cb0"
server_key_version: "encryptionkey@0"
I20260928 16:39:10.248430 26785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 16:39:10.252396 26800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:10.254516 26785 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:10.254808 26785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "136c5537dc544e6e8b12cf0beb5b45da"
format_stamp: "Formatted at 2026-09-28 16:39:10 on dist-test-slave-nqgh"
server_key: "db6ef58572275e6d7843e8d70b3c1337"
server_key_iv: "aa55495ad9e1c158e8c7ceff479d8cb0"
server_key_version: "encryptionkey@0"
I20260928 16:39:10.255012 26785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:10.268931 26785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:10.273864 26785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:10.274233 26785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:10.275174 26785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:10.276340 26785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:10.276397 26785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:10.276465 26785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:10.276489 26785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:10.288528 26785 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:37005
I20260928 16:39:10.288558 26913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:37005 every 8 connection(s)
I20260928 16:39:10.289711 26785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:39:10.298446 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 26785
I20260928 16:39:10.298601 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:39:10.298913 25885 external_mini_cluster.cc:1428] Setting key f144dfaf580d74475269c2fd2116391d
Sep 28 16:39:10 dist-test-slave-nqgh krb5kdc[26410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613550, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:10.307243 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:10.292134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:50535 (local address 127.25.71.126:45165)
0928 16:39:10.292322 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:10.292327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:10.293537 (+ 1210us) server_negotiation.cc:407] Connection header received
0928 16:39:10.294762 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:10.294767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:10.294862 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:10.294998 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:10.296517 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.297173 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.298836 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.299191 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.299326 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:10.302835 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:10.302839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:10.302841 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:10.302886 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:10.304518 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:10.305280 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:10.305284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:10.305285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:10.305403 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:10.305858 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:10.305861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:10.305862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:10.306125 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:10.306259 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:10.306903 (+ 644us) server_negotiation.cc:299] Negotiation successful
0928 16:39:10.307063 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 16:39:10.308208 26916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:10.292679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:45165 (local address 127.25.71.67:50535)
0928 16:39:10.293315 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:10.293375 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:10.294393 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:10.295200 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:10.295208 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:10.295662 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:10.296327 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:10.296343 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.297346 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.297349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:10.298573 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:10.298587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.299412 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.299417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:10.299610 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:10.300362 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:10.300381 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:10.302517 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:10.304704 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:10.304711 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:10.304729 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:10.305095 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:10.305552 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:10.305555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:10.305556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:10.305748 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:10.306245 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:10.306251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:10.306571 (+ 320us) client_negotiation.cc:769] Sending connection context
0928 16:39:10.306822 (+ 251us) client_negotiation.cc:240] Negotiation successful
0928 16:39:10.307158 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":148,"threads_started":1}
I20260928 16:39:10.309865 26914 heartbeater.cc:347] Connected to a master server at 127.25.71.126:45165
I20260928 16:39:10.310189 26914 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:10.310812 26914 heartbeater.cc:511] Master 127.25.71.126:45165 requested a full tablet report, sending...
I20260928 16:39:10.312182 26455 ts_manager.cc:194] Registered new tserver with Master: 136c5537dc544e6e8b12cf0beb5b45da (127.25.71.67:37005)
I20260928 16:39:10.312978 26455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:50535
I20260928 16:39:10.314321 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:39:10.328109 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:10.318930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.25.71.126:45165)
0928 16:39:10.319135 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:10.319141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:10.319362 (+ 221us) server_negotiation.cc:407] Connection header received
0928 16:39:10.319515 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:10.319517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:10.319579 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:10.319689 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:10.320556 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.321181 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.322268 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.322523 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.322711 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:10.324270 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:10.324273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:10.324276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:10.324308 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:10.325942 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:10.326538 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:10.326543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:10.326544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:10.326705 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:10.327047 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:10.327051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:10.327053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:10.327372 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:10.327512 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:10.327786 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 16:39:10.327936 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:39:10.335062 26455 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:39:10.338828 26455 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:39:10.357754 26925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:10.350721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:39371 (local address 127.0.0.1:37706)
0928 16:39:10.351378 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:10.351447 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:10.351641 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:10.353079 (+ 1438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:10.353086 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:10.353114 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:10.353318 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:10.353326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.354528 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.354532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:10.355905 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:10.355916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.357055 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.357059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:10.357216 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:10.357271 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 16:39:10.357403 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 16:39:10.357568 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":132,"threads_started":1}
I20260928 16:39:10.357991 26927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:10.351466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:44253 (local address 127.0.0.1:45664)
0928 16:39:10.352290 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:10.352305 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:10.352454 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:10.353461 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:10.353463 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:10.353485 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:10.353589 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:10.353595 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.354990 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.354993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:10.356268 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:10.356281 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.357444 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.357448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:10.357569 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:10.357596 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:39:10.357670 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 16:39:10.357778 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":31,"thread_start_us":123,"threads_started":1}
I20260928 16:39:10.358014 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:10.351936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45664 (local address 127.25.71.65:44253)
0928 16:39:10.352268 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:10.352276 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:10.352555 (+ 279us) server_negotiation.cc:407] Connection header received
0928 16:39:10.352637 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:10.352643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:10.352886 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:10.353056 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:10.353725 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.354875 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.356445 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.357307 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.357418 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:10.357563 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:10.357720 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 16:39:10.357847 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":89,"threads_started":1}
I20260928 16:39:10.358600 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:10.350942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37706 (local address 127.25.71.66:39371)
0928 16:39:10.352460 (+ 1518us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:10.352475 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:10.352506 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:39:10.352602 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:10.352609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:10.352886 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:10.353094 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:10.353438 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.354379 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.356063 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.356898 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.357026 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:10.357230 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:10.358254 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0928 16:39:10.358419 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1388,"thread_start_us":151,"threads_started":1}
I20260928 16:39:10.359009 26928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:10.350989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:37005 (local address 127.0.0.1:33658)
0928 16:39:10.351914 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:10.351929 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:10.352018 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:10.353238 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:10.353241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:10.353261 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:10.353388 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:10.353394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.354960 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.354964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:10.356530 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:10.356538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.357822 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.357826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:10.358487 (+ 661us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:10.358516 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 16:39:10.358591 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 16:39:10.358876 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"mutex_wait_us":264,"thread_start_us":91,"threads_started":1}
I20260928 16:39:10.359787 26929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:10.351816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33658 (local address 127.25.71.67:37005)
0928 16:39:10.352515 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:10.352525 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:10.352551 (+ 26us) server_negotiation.cc:407] Connection header received
0928 16:39:10.352636 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:10.352643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:10.352886 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:10.353056 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:10.353512 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.354761 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.356652 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.357500 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.357594 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:10.357754 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:10.359487 (+ 1733us) server_negotiation.cc:299] Negotiation successful
0928 16:39:10.359637 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":115,"threads_started":1}
I20260928 16:39:10.361979 26580 tablet_service.cc:1467] Processing CreateTablet for tablet a37b83930ea94568b906a9123a3d1e07 (DEFAULT_TABLE table=test-table [id=93251f5934ea46c8a5c5dcde41a18d29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:10.362206 26714 tablet_service.cc:1467] Processing CreateTablet for tablet a37b83930ea94568b906a9123a3d1e07 (DEFAULT_TABLE table=test-table [id=93251f5934ea46c8a5c5dcde41a18d29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:10.362872 26848 tablet_service.cc:1467] Processing CreateTablet for tablet a37b83930ea94568b906a9123a3d1e07 (DEFAULT_TABLE table=test-table [id=93251f5934ea46c8a5c5dcde41a18d29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:10.363476 26714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37b83930ea94568b906a9123a3d1e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:10.363543 26580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37b83930ea94568b906a9123a3d1e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:10.363905 26848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37b83930ea94568b906a9123a3d1e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:10.370571 26931 tablet_bootstrap.cc:492] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933: Bootstrap starting.
I20260928 16:39:10.373653 26931 tablet_bootstrap.cc:654] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:10.373814 26932 tablet_bootstrap.cc:492] T a37b83930ea94568b906a9123a3d1e07 P 425cfc8675214eedb1ed4aaece127294: Bootstrap starting.
I20260928 16:39:10.374002 26933 tablet_bootstrap.cc:492] T a37b83930ea94568b906a9123a3d1e07 P 136c5537dc544e6e8b12cf0beb5b45da: Bootstrap starting.
I20260928 16:39:10.374679 26931 log.cc:824] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:10.376502 26933 tablet_bootstrap.cc:654] T a37b83930ea94568b906a9123a3d1e07 P 136c5537dc544e6e8b12cf0beb5b45da: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:10.376593 26932 tablet_bootstrap.cc:654] T a37b83930ea94568b906a9123a3d1e07 P 425cfc8675214eedb1ed4aaece127294: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:10.377561 26931 tablet_bootstrap.cc:492] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933: No bootstrap required, opened a new log
I20260928 16:39:10.377580 26932 log.cc:824] T a37b83930ea94568b906a9123a3d1e07 P 425cfc8675214eedb1ed4aaece127294: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:10.377825 26931 ts_tablet_manager.cc:1397] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 16:39:10.377880 26933 log.cc:824] T a37b83930ea94568b906a9123a3d1e07 P 136c5537dc544e6e8b12cf0beb5b45da: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:10.380340 26932 tablet_bootstrap.cc:492] T a37b83930ea94568b906a9123a3d1e07 P 425cfc8675214eedb1ed4aaece127294: No bootstrap required, opened a new log
I20260928 16:39:10.380620 26932 ts_tablet_manager.cc:1397] T a37b83930ea94568b906a9123a3d1e07 P 425cfc8675214eedb1ed4aaece127294: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 16:39:10.380717 26933 tablet_bootstrap.cc:492] T a37b83930ea94568b906a9123a3d1e07 P 136c5537dc544e6e8b12cf0beb5b45da: No bootstrap required, opened a new log
I20260928 16:39:10.380968 26933 ts_tablet_manager.cc:1397] T a37b83930ea94568b906a9123a3d1e07 P 136c5537dc544e6e8b12cf0beb5b45da: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 16:39:10.382157 26931 raft_consensus.cc:348] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41ef3c8066f5425f86de9300982b3933" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 39371 } } peers { permanent_uuid: "136c5537dc544e6e8b12cf0beb5b45da" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 37005 } } peers { permanent_uuid: "425cfc8675214eedb1ed4aaece127294" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 44253 } }
I20260928 16:39:10.382489 26931 raft_consensus.cc:374] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:10.382557 26931 raft_consensus.cc:729] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41ef3c8066f5425f86de9300982b3933, State: Initialized, Role: FOLLOWER
I20260928 16:39:10.383190 26931 consensus_queue.cc:260] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41ef3c8066f5425f86de9300982b3933" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 39371 } } peers { permanent_uuid: "136c5537dc544e6e8b12cf0beb5b45da" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 37005 } } peers { permanent_uuid: "425cfc8675214eedb1ed4aaece127294" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 44253 } }
I20260928 16:39:10.384346 26780 heartbeater.cc:503] Master 127.25.71.126:45165 was elected leader, sending a full tablet report...
I20260928 16:39:10.384465 26931 ts_tablet_manager.cc:1428] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 16:39:10.385248 26933 raft_consensus.cc:348] T a37b83930ea94568b906a9123a3d1e07 P 136c5537dc544e6e8b12cf0beb5b45da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41ef3c8066f5425f86de9300982b3933" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 39371 } } peers { permanent_uuid: "136c5537dc544e6e8b12cf0beb5b45da" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 37005 } } peers { permanent_uuid: "425cfc8675214eedb1ed4aaece127294" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 44253 } }
I20260928 16:39:10.385517 26933 raft_consensus.cc:374] T a37b83930ea94568b906a9123a3d1e07 P 136c5537dc544e6e8b12cf0beb5b45da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:10.385571 26933 raft_consensus.cc:729] T a37b83930ea94568b906a9123a3d1e07 P 136c5537dc544e6e8b12cf0beb5b45da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136c5537dc544e6e8b12cf0beb5b45da, State: Initialized, Role: FOLLOWER
I20260928 16:39:10.385658 26932 raft_consensus.cc:348] T a37b83930ea94568b906a9123a3d1e07 P 425cfc8675214eedb1ed4aaece127294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41ef3c8066f5425f86de9300982b3933" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 39371 } } peers { permanent_uuid: "136c5537dc544e6e8b12cf0beb5b45da" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 37005 } } peers { permanent_uuid: "425cfc8675214eedb1ed4aaece127294" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 44253 } }
I20260928 16:39:10.386003 26932 raft_consensus.cc:374] T a37b83930ea94568b906a9123a3d1e07 P 425cfc8675214eedb1ed4aaece127294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:10.386068 26932 raft_consensus.cc:729] T a37b83930ea94568b906a9123a3d1e07 P 425cfc8675214eedb1ed4aaece127294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 425cfc8675214eedb1ed4aaece127294, State: Initialized, Role: FOLLOWER
I20260928 16:39:10.386047 26933 consensus_queue.cc:260] T a37b83930ea94568b906a9123a3d1e07 P 136c5537dc544e6e8b12cf0beb5b45da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41ef3c8066f5425f86de9300982b3933" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 39371 } } peers { permanent_uuid: "136c5537dc544e6e8b12cf0beb5b45da" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 37005 } } peers { permanent_uuid: "425cfc8675214eedb1ed4aaece127294" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 44253 } }
I20260928 16:39:10.386658 26932 consensus_queue.cc:260] T a37b83930ea94568b906a9123a3d1e07 P 425cfc8675214eedb1ed4aaece127294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41ef3c8066f5425f86de9300982b3933" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 39371 } } peers { permanent_uuid: "136c5537dc544e6e8b12cf0beb5b45da" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 37005 } } peers { permanent_uuid: "425cfc8675214eedb1ed4aaece127294" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 44253 } }
I20260928 16:39:10.387296 26914 heartbeater.cc:503] Master 127.25.71.126:45165 was elected leader, sending a full tablet report...
I20260928 16:39:10.387840 26933 ts_tablet_manager.cc:1428] T a37b83930ea94568b906a9123a3d1e07 P 136c5537dc544e6e8b12cf0beb5b45da: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 16:39:10.387907 26932 ts_tablet_manager.cc:1428] T a37b83930ea94568b906a9123a3d1e07 P 425cfc8675214eedb1ed4aaece127294: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 16:39:10.388312 26646 heartbeater.cc:503] Master 127.25.71.126:45165 was elected leader, sending a full tablet report...
I20260928 16:39:10.406704 26937 raft_consensus.cc:482] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:39:10.407042 26937 raft_consensus.cc:504] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41ef3c8066f5425f86de9300982b3933" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 39371 } } peers { permanent_uuid: "136c5537dc544e6e8b12cf0beb5b45da" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 37005 } } peers { permanent_uuid: "425cfc8675214eedb1ed4aaece127294" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 44253 } }
I20260928 16:39:10.408737 26937 leader_election.cc:288] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 136c5537dc544e6e8b12cf0beb5b45da (127.25.71.67:37005), 425cfc8675214eedb1ed4aaece127294 (127.25.71.65:44253)
I20260928 16:39:10.414712 26782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:10.409171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:44253 (local address 127.25.71.66:54625)
0928 16:39:10.409674 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:10.409690 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:10.409830 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:10.410556 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:10.410559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:10.410576 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:10.410709 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:10.410717 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.411612 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.411617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:10.413030 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:10.413044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.414144 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.414148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:10.414285 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:10.414313 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 16:39:10.414392 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 16:39:10.414510 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":13,"thread_start_us":172,"threads_started":1}
I20260928 16:39:10.414712 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:10.409460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:54625 (local address 127.25.71.65:44253)
0928 16:39:10.409597 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:10.409602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:10.409808 (+ 206us) server_negotiation.cc:407] Connection header received
0928 16:39:10.410360 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:10.410363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:10.410420 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:10.410498 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:10.410849 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.411476 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.413205 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.413831 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.413997 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:10.414068 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:10.414427 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 16:39:10.414523 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:39:10.414948 26929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:10.409195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:52919 (local address 127.25.71.67:37005)
0928 16:39:10.409352 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:10.409357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:10.409781 (+ 424us) server_negotiation.cc:407] Connection header received
0928 16:39:10.410000 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:10.410005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:10.410088 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:10.410196 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:10.410758 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.411588 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.413419 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.414077 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.414210 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:10.414264 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:10.414591 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 16:39:10.414769 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:39:10.415473 26940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:10.409060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:37005 (local address 127.25.71.66:52919)
0928 16:39:10.409674 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:10.409690 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:10.409835 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:10.410328 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:10.410331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:10.410354 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:10.410506 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:10.410514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.411723 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.411725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:10.413211 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:10.413234 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.414242 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.414246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:10.414407 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:10.414443 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 16:39:10.415162 (+ 719us) client_negotiation.cc:240] Negotiation successful
0928 16:39:10.415319 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":64}
I20260928 16:39:10.416082 26600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37b83930ea94568b906a9123a3d1e07" candidate_uuid: "41ef3c8066f5425f86de9300982b3933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "425cfc8675214eedb1ed4aaece127294" is_pre_election: true
I20260928 16:39:10.416479 26600 raft_consensus.cc:2454] T a37b83930ea94568b906a9123a3d1e07 P 425cfc8675214eedb1ed4aaece127294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41ef3c8066f5425f86de9300982b3933 in term 0.
I20260928 16:39:10.416476 26868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37b83930ea94568b906a9123a3d1e07" candidate_uuid: "41ef3c8066f5425f86de9300982b3933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136c5537dc544e6e8b12cf0beb5b45da" is_pre_election: true
I20260928 16:39:10.416900 26868 raft_consensus.cc:2454] T a37b83930ea94568b906a9123a3d1e07 P 136c5537dc544e6e8b12cf0beb5b45da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41ef3c8066f5425f86de9300982b3933 in term 0.
I20260928 16:39:10.417098 26670 leader_election.cc:302] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41ef3c8066f5425f86de9300982b3933, 425cfc8675214eedb1ed4aaece127294; no voters:
I20260928 16:39:10.417805 26937 raft_consensus.cc:2781] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:39:10.417903 26937 raft_consensus.cc:482] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:39:10.417933 26937 raft_consensus.cc:3037] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:10.419339 26937 raft_consensus.cc:504] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41ef3c8066f5425f86de9300982b3933" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 39371 } } peers { permanent_uuid: "136c5537dc544e6e8b12cf0beb5b45da" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 37005 } } peers { permanent_uuid: "425cfc8675214eedb1ed4aaece127294" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 44253 } }
I20260928 16:39:10.419837 26937 leader_election.cc:288] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933 [CANDIDATE]: Term 1 election: Requested vote from peers 136c5537dc544e6e8b12cf0beb5b45da (127.25.71.67:37005), 425cfc8675214eedb1ed4aaece127294 (127.25.71.65:44253)
I20260928 16:39:10.420446 26600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37b83930ea94568b906a9123a3d1e07" candidate_uuid: "41ef3c8066f5425f86de9300982b3933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "425cfc8675214eedb1ed4aaece127294"
I20260928 16:39:10.420622 26600 raft_consensus.cc:3037] T a37b83930ea94568b906a9123a3d1e07 P 425cfc8675214eedb1ed4aaece127294 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:10.422323 26600 raft_consensus.cc:2454] T a37b83930ea94568b906a9123a3d1e07 P 425cfc8675214eedb1ed4aaece127294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41ef3c8066f5425f86de9300982b3933 in term 1.
I20260928 16:39:10.422833 26868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37b83930ea94568b906a9123a3d1e07" candidate_uuid: "41ef3c8066f5425f86de9300982b3933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136c5537dc544e6e8b12cf0beb5b45da"
I20260928 16:39:10.422916 26670 leader_election.cc:302] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41ef3c8066f5425f86de9300982b3933, 425cfc8675214eedb1ed4aaece127294; no voters:
I20260928 16:39:10.423017 26868 raft_consensus.cc:3037] T a37b83930ea94568b906a9123a3d1e07 P 136c5537dc544e6e8b12cf0beb5b45da [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:10.423226 26937 raft_consensus.cc:2781] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:10.423547 26937 raft_consensus.cc:686] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933 [term 1 LEADER]: Becoming Leader. State: Replica: 41ef3c8066f5425f86de9300982b3933, State: Running, Role: LEADER
I20260928 16:39:10.424075 26937 consensus_queue.cc:237] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41ef3c8066f5425f86de9300982b3933" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 39371 } } peers { permanent_uuid: "136c5537dc544e6e8b12cf0beb5b45da" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 37005 } } peers { permanent_uuid: "425cfc8675214eedb1ed4aaece127294" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 44253 } }
I20260928 16:39:10.424360 26868 raft_consensus.cc:2454] T a37b83930ea94568b906a9123a3d1e07 P 136c5537dc544e6e8b12cf0beb5b45da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41ef3c8066f5425f86de9300982b3933 in term 1.
I20260928 16:39:10.428690 26454 catalog_manager.cc:5495] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41ef3c8066f5425f86de9300982b3933 (127.25.71.66). New cstate: current_term: 1 leader_uuid: "41ef3c8066f5425f86de9300982b3933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41ef3c8066f5425f86de9300982b3933" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 39371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "136c5537dc544e6e8b12cf0beb5b45da" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 37005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "425cfc8675214eedb1ed4aaece127294" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 44253 } health_report { overall_health: UNKNOWN } } }
Sep 28 16:39:10 dist-test-slave-nqgh krb5kdc[26410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.65@KRBTEST.COM
I20260928 16:39:10.450601 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:10.439422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45672 (local address 127.25.71.65:44253)
0928 16:39:10.439662 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:10.439666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:10.439877 (+ 211us) server_negotiation.cc:407] Connection header received
0928 16:39:10.439955 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:10.439957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:10.440017 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:10.440104 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:10.440956 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.441531 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.442693 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.443006 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.443105 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:10.446289 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:10.446308 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:10.446327 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:10.446364 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:10.448215 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:10.448944 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:10.448950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:10.448952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:10.449055 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:10.449443 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:10.449446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:10.449447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:10.449749 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:10.449953 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:10.450262 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 16:39:10.450424 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
Sep 28 16:39:10 dist-test-slave-nqgh krb5kdc[26410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613550, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:39:10 dist-test-slave-nqgh krb5kdc[26410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613550, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.71.65@KRBTEST.COM
I20260928 16:39:10.483539 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:10.474051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45688 (local address 127.25.71.65:44253)
0928 16:39:10.474278 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:10.474283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:10.474428 (+ 145us) server_negotiation.cc:407] Connection header received
0928 16:39:10.474594 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:10.474597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:10.474676 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:10.474752 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:10.475553 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.476100 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.476956 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.477233 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.477331 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:10.480313 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:10.480317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:10.480320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:10.480352 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:10.481687 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:10.482213 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:10.482216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:10.482218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:10.482298 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:10.482545 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:10.482548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:10.482550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:10.482803 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:10.482924 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:10.483254 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 16:39:10.483382 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
Sep 28 16:39:10 dist-test-slave-nqgh krb5kdc[26410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:39:10 dist-test-slave-nqgh krb5kdc[26410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.65@KRBTEST.COM
W20260928 16:39:10.512264 26647 tablet.cc:2369] T a37b83930ea94568b906a9123a3d1e07 P 425cfc8675214eedb1ed4aaece127294: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:39:10.516302 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:10.504806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45696 (local address 127.25.71.65:44253)
0928 16:39:10.505003 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:10.505007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:10.505251 (+ 244us) server_negotiation.cc:407] Connection header received
0928 16:39:10.505378 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:10.505387 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:10.505450 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:10.505571 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:10.506560 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.507190 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.508280 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:10.508528 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:10.508634 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:10.512042 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:10.512048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:10.512051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:10.512100 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:10.513962 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:10.514770 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:10.514773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:10.514774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:10.514890 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:10.515207 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:10.515210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:10.515211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:10.515476 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:10.515591 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:10.515971 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 16:39:10.516164 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260928 16:39:10.518167 26560 tablet_service.cc:2908] Rejecting scan request for tablet a37b83930ea94568b906a9123a3d1e07: Uninitialized: clean time has not yet been initialized
I20260928 16:39:10.521070 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 26517
W20260928 16:39:10.521409 26781 tablet.cc:2369] T a37b83930ea94568b906a9123a3d1e07 P 41ef3c8066f5425f86de9300982b3933: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:39:10.528096 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 26651
I20260928 16:39:10.533622 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 26785
I20260928 16:39:10.540997 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 26423
2026-09-28T16:39:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2482 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:39:10 dist-test-slave-nqgh krb5kdc[26955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:39:10 dist-test-slave-nqgh krb5kdc[26955](info): set up 2 sockets
Sep 28 16:39:10 dist-test-slave-nqgh krb5kdc[26955](info): commencing operation
krb5kdc: starting...
W20260928 16:39:11.516284 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:39:11 dist-test-slave-nqgh krb5kdc[26955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:39:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:39:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:39:11.663015 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:44207
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:42879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:44207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:11.804814 26968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:11.805255 26968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:11.805321 26968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:11.809976 26968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:11.810077 26968 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 16:39:11.810104 26968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:11.810132 26968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:11.810153 26968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:11.810271 26968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:39:11.817034 26968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:42879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:44207
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:44207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:11.818697 26968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:11.820127 26968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:39:11.831385 26968 server_base.cc:1034] running on GCE node
W20260928 16:39:11.831321 26973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:11.831300 26974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:11.831321 26976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:11.832553 26968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:11.834170 26968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:11.835389 26968 hybrid_clock.cc:648] HybridClock initialized: now 1790613551835354 us; error 52 us; skew 500 ppm
Sep 28 16:39:11 dist-test-slave-nqgh krb5kdc[26955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613551, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:11.845541 26968 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:39:11.847081 26968 webserver.cc:460] Webserver started at http://127.25.71.126:45345/ using document root <none> and password file <none>
I20260928 16:39:11.847846 26968 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:11.847926 26968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:11.848198 26968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:11.851047 26968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "6f61f45b2b9f44faa0b09780f282feb9"
format_stamp: "Formatted at 2026-09-28 16:39:11 on dist-test-slave-nqgh"
server_key: "851e07bec5f8f42ee165313a51b317d6"
server_key_iv: "1e4671f799a580149b57cdb2020386a7"
server_key_version: "encryptionkey@0"
I20260928 16:39:11.851809 26968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "6f61f45b2b9f44faa0b09780f282feb9"
format_stamp: "Formatted at 2026-09-28 16:39:11 on dist-test-slave-nqgh"
server_key: "851e07bec5f8f42ee165313a51b317d6"
server_key_iv: "1e4671f799a580149b57cdb2020386a7"
server_key_version: "encryptionkey@0"
I20260928 16:39:11.857229 26968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:39:11.861389 26983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:11.863628 26968 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 16:39:11.863847 26968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "6f61f45b2b9f44faa0b09780f282feb9"
format_stamp: "Formatted at 2026-09-28 16:39:11 on dist-test-slave-nqgh"
server_key: "851e07bec5f8f42ee165313a51b317d6"
server_key_iv: "1e4671f799a580149b57cdb2020386a7"
server_key_version: "encryptionkey@0"
I20260928 16:39:11.864037 26968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:11.886909 26968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:11.891108 26968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:11.891330 26968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:11.902246 26968 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:44207
I20260928 16:39:11.902390 27045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:44207 every 8 connection(s)
I20260928 16:39:11.903681 26968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:39:11.907662 27046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:11.912952 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 26968
I20260928 16:39:11.913100 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:39:11.913380 25885 external_mini_cluster.cc:1428] Setting key af342d94efd2de04cb4f1b107b993dfc
I20260928 16:39:11.914423 27046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9: Bootstrap starting.
I20260928 16:39:11.917249 27046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:11.918239 27046 log.cc:824] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:11.921332 27046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9: No bootstrap required, opened a new log
Sep 28 16:39:11 dist-test-slave-nqgh krb5kdc[26955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:11.926136 27046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f61f45b2b9f44faa0b09780f282feb9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44207 } }
I20260928 16:39:11.926482 27046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:11.926538 27046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f61f45b2b9f44faa0b09780f282feb9, State: Initialized, Role: FOLLOWER
I20260928 16:39:11.927090 27046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f61f45b2b9f44faa0b09780f282feb9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44207 } }
I20260928 16:39:11.927237 27046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:39:11.927316 27046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:39:11.927429 27046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:11.928748 27046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f61f45b2b9f44faa0b09780f282feb9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44207 } }
I20260928 16:39:11.929143 27046 leader_election.cc:302] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f61f45b2b9f44faa0b09780f282feb9; no voters:
I20260928 16:39:11.929659 27046 leader_election.cc:288] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:39:11.929875 27051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:11.930384 27051 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9 [term 1 LEADER]: Becoming Leader. State: Replica: 6f61f45b2b9f44faa0b09780f282feb9, State: Running, Role: LEADER
I20260928 16:39:11.930959 27051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f61f45b2b9f44faa0b09780f282feb9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44207 } }
I20260928 16:39:11.931406 27046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:11.933099 27052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f61f45b2b9f44faa0b09780f282feb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f61f45b2b9f44faa0b09780f282feb9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44207 } } }
I20260928 16:39:11.933287 27052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:11.933715 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:11.915349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.25.71.126:44207)
0928 16:39:11.915923 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:11.915943 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:11.915994 (+ 51us) server_negotiation.cc:407] Connection header received
0928 16:39:11.916872 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:11.916897 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:11.917367 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:11.917825 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:11.918909 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:11.920355 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:11.921717 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:11.922420 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:11.922839 (+ 419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:11.926100 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:11.926109 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:11.926136 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:11.926190 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:11.928542 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:11.929266 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:11.929274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:11.929281 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:11.929456 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:11.930019 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:11.930023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:11.930025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:11.930687 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:11.931678 (+ 991us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:11.931978 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 16:39:11.932359 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":171,"threads_started":1}
I20260928 16:39:11.934422 27064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:39:11.933699 27053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f61f45b2b9f44faa0b09780f282feb9. Latest consensus state: current_term: 1 leader_uuid: "6f61f45b2b9f44faa0b09780f282feb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f61f45b2b9f44faa0b09780f282feb9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44207 } } }
I20260928 16:39:11.934578 27053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:11.939204 27064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:39:11.945827 27064 catalog_manager.cc:1347] Generated new cluster ID: 9176e2aa238c48d290e354dcc9d99592
I20260928 16:39:11.945940 27064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:39:11.963565 27064 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:39:11.964780 27064 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:39:11.985146 27064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9: Generated new TSK 0
I20260928 16:39:11.986017 27064 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
I20260928 16:39:12.031903 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:44207
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:42879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:12.197147 27072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:12.197499 27072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:12.197549 27072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:12.202282 27072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:12.202406 27072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:12.202454 27072 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:39:12.202488 27072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:39:12.209510 27072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:42879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.71.126:44207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:12.210916 27072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:12.212250 27072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:12.225446 27080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:12.225446 27078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:12.225860 27072 server_base.cc:1034] running on GCE node
W20260928 16:39:12.225675 27077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:12.226599 27072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:12.227650 27072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:12.228955 27072 hybrid_clock.cc:648] HybridClock initialized: now 1790613552228934 us; error 94 us; skew 500 ppm
Sep 28 16:39:12 dist-test-slave-nqgh krb5kdc[26955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613552, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:12.233278 27072 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:39:12.234951 27072 webserver.cc:460] Webserver started at http://127.25.71.65:43893/ using document root <none> and password file <none>
I20260928 16:39:12.235767 27072 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:12.235867 27072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:12.236169 27072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:12.239136 27072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "822bbcfb122f43528736000f607ffdfd"
format_stamp: "Formatted at 2026-09-28 16:39:12 on dist-test-slave-nqgh"
server_key: "7aeafa1eb728f5d6f8156f0bee40a705"
server_key_iv: "2f50786e70e0063bad9063ea8a621db3"
server_key_version: "encryptionkey@0"
I20260928 16:39:12.239918 27072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "822bbcfb122f43528736000f607ffdfd"
format_stamp: "Formatted at 2026-09-28 16:39:12 on dist-test-slave-nqgh"
server_key: "7aeafa1eb728f5d6f8156f0bee40a705"
server_key_iv: "2f50786e70e0063bad9063ea8a621db3"
server_key_version: "encryptionkey@0"
I20260928 16:39:12.245434 27072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:39:12.249354 27087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:12.251483 27072 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:12.251710 27072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "822bbcfb122f43528736000f607ffdfd"
format_stamp: "Formatted at 2026-09-28 16:39:12 on dist-test-slave-nqgh"
server_key: "7aeafa1eb728f5d6f8156f0bee40a705"
server_key_iv: "2f50786e70e0063bad9063ea8a621db3"
server_key_version: "encryptionkey@0"
I20260928 16:39:12.251914 27072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:12.264321 27072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:12.269379 27072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:12.269668 27072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:12.271755 27072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:12.271822 27072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:12.271885 27072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:12.271916 27072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:39:12 dist-test-slave-nqgh krb5kdc[26955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613552, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:12.285156 27072 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:39791
I20260928 16:39:12.285195 27202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:39791 every 8 connection(s)
I20260928 16:39:12.286517 27072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:39:12.291097 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.272929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:56691 (local address 127.25.71.126:44207)
0928 16:39:12.273135 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:12.273139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:12.274746 (+ 1607us) server_negotiation.cc:407] Connection header received
0928 16:39:12.276202 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:12.276206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:12.276277 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:12.276370 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:12.278952 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.279766 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.281820 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.282114 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.282337 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.285774 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:12.285779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:12.285782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:12.285815 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:12.287717 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:12.288715 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:12.288719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:12.288720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:12.288836 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:12.289503 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:12.289507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:12.289509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:12.289750 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:12.289861 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:12.290700 (+ 839us) server_negotiation.cc:299] Negotiation successful
0928 16:39:12.290952 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260928 16:39:12.292244 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.273697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44207 (local address 127.25.71.65:56691)
0928 16:39:12.274484 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:12.274571 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:12.275945 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:12.276838 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:12.276852 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:12.277541 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:12.278615 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.278686 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.279964 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.279969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.281421 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.281625 (+ 204us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.282392 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.282397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.282706 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.283530 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:12.283554 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:12.285520 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:12.287915 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:12.287922 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:12.287944 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:12.288466 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:12.289029 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:12.289032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:12.289034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:12.289316 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:12.289922 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:12.289930 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:12.290320 (+ 390us) client_negotiation.cc:769] Sending connection context
0928 16:39:12.290541 (+ 221us) client_negotiation.cc:240] Negotiation successful
0928 16:39:12.291034 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":215,"threads_started":1}
I20260928 16:39:12.292725 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 27072
I20260928 16:39:12.292851 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:39:12.293223 25885 external_mini_cluster.cc:1428] Setting key 50c0d0349d02dffcd23f4521c46a8d2f
I20260928 16:39:12.294715 27203 heartbeater.cc:347] Connected to a master server at 127.25.71.126:44207
I20260928 16:39:12.294941 27203 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:12.295684 27203 heartbeater.cc:511] Master 127.25.71.126:44207 requested a full tablet report, sending...
I20260928 16:39:12.297799 26999 ts_manager.cc:194] Registered new tserver with Master: 822bbcfb122f43528736000f607ffdfd (127.25.71.65:39791)
I20260928 16:39:12.299434 26999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:56691
WARNING: no policy specified for kudu/127.25.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.66@KRBTEST.COM" created.
I20260928 16:39:12.308317 27210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.297108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44207 (local address 127.25.71.65:53979)
0928 16:39:12.298874 (+ 1766us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:12.298891 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:12.299034 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:12.299630 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:12.299634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:12.299898 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:12.300070 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.300080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.301174 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.301178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.301831 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.301842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.302341 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.302346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.302506 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.303192 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:12.303208 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:12.303715 (+ 507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:12.306145 (+ 2430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:12.306148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:12.306157 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:12.306400 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:12.306841 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:12.306844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:12.306846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:12.307015 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:12.307617 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:12.307620 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:12.307770 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 16:39:12.307904 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 16:39:12.308119 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1645,"mutex_wait_us":48,"thread_start_us":151,"threads_started":1}
I20260928 16:39:12.308367 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.297205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:53979 (local address 127.25.71.126:44207)
0928 16:39:12.297378 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:12.297383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:12.299004 (+ 1621us) server_negotiation.cc:407] Connection header received
0928 16:39:12.299150 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:12.299153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:12.299213 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:12.299331 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:12.300348 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.301017 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.301963 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.302209 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.302300 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.303890 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:12.303894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:12.303897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:12.303925 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:12.305928 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:12.306518 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:12.306522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:12.306524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:12.306656 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:12.307170 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:12.307173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:12.307175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:12.307457 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:12.307575 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:12.307990 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 16:39:12.308167 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
I20260928 16:39:12.327855 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:44207
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:42879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:12.469877 27212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:12.470270 27212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:12.470325 27212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:12.474933 27212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:12.475046 27212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:12.475091 27212 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:39:12.475129 27212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:39:12.481628 27212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:42879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.71.126:44207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:12.483189 27212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:12.484642 27212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:12.497187 27218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:12.497187 27217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:12.497195 27220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:12.497953 27212 server_base.cc:1034] running on GCE node
I20260928 16:39:12.498581 27212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:12.499509 27212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:12.500733 27212 hybrid_clock.cc:648] HybridClock initialized: now 1790613552500693 us; error 56 us; skew 500 ppm
Sep 28 16:39:12 dist-test-slave-nqgh krb5kdc[26955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613552, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:12.504801 27212 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:39:12.506325 27212 webserver.cc:460] Webserver started at http://127.25.71.66:45539/ using document root <none> and password file <none>
I20260928 16:39:12.507198 27212 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:12.507295 27212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:12.507566 27212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:12.510421 27212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "762e1920adbd485d87832a7faa7ed2d0"
format_stamp: "Formatted at 2026-09-28 16:39:12 on dist-test-slave-nqgh"
server_key: "534b1e923fdf930173b2a52bcde4cee1"
server_key_iv: "8c39c23cffa468f9cb23618caa96903a"
server_key_version: "encryptionkey@0"
I20260928 16:39:12.511278 27212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "762e1920adbd485d87832a7faa7ed2d0"
format_stamp: "Formatted at 2026-09-28 16:39:12 on dist-test-slave-nqgh"
server_key: "534b1e923fdf930173b2a52bcde4cee1"
server_key_iv: "8c39c23cffa468f9cb23618caa96903a"
server_key_version: "encryptionkey@0"
I20260928 16:39:12.516961 27212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:39:12.521179 27227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:12.523495 27212 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:39:12.523718 27212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "762e1920adbd485d87832a7faa7ed2d0"
format_stamp: "Formatted at 2026-09-28 16:39:12 on dist-test-slave-nqgh"
server_key: "534b1e923fdf930173b2a52bcde4cee1"
server_key_iv: "8c39c23cffa468f9cb23618caa96903a"
server_key_version: "encryptionkey@0"
I20260928 16:39:12.523907 27212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:12.552350 27212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:12.556759 27212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:12.557139 27212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:12.559353 27212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:12.559423 27212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:12.559473 27212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:12.559487 27212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:39:12 dist-test-slave-nqgh krb5kdc[26955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613552, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:12.571928 27212 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:33913
I20260928 16:39:12.572221 27342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:33913 every 8 connection(s)
I20260928 16:39:12.573243 27212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:39:12.576004 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.560427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:37901 (local address 127.25.71.126:44207)
0928 16:39:12.560640 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:12.560647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:12.561626 (+ 979us) server_negotiation.cc:407] Connection header received
0928 16:39:12.562829 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:12.562834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:12.562909 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:12.562970 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:12.564720 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.565713 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.567411 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.567737 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.567863 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.571431 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:12.571434 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:12.571437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:12.571466 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:12.573234 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:12.574091 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:12.574097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:12.574099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:12.574235 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:12.574662 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:12.574666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:12.574668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:12.574918 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:12.575017 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:12.575619 (+ 602us) server_negotiation.cc:299] Negotiation successful
0928 16:39:12.575787 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:39:12.577083 27236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.560848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44207 (local address 127.25.71.66:37901)
0928 16:39:12.561427 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:12.561483 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:12.562462 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:12.563392 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:12.563399 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:12.563808 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:12.564492 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.564515 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.565970 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.565975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.567226 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.567243 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.567934 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.567938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.568177 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.569521 (+ 1344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:12.569543 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:12.571193 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:12.573416 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:12.573423 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:12.573443 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:12.573910 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:12.574339 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:12.574343 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:12.574345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:12.574520 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:12.575037 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:12.575045 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:12.575341 (+ 296us) client_negotiation.cc:769] Sending connection context
0928 16:39:12.575534 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 16:39:12.575956 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":161,"threads_started":1}
I20260928 16:39:12.578107 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 27212
I20260928 16:39:12.578227 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:39:12.578586 25885 external_mini_cluster.cc:1428] Setting key 796134b815f5b92b59988f01e7cee4cb
I20260928 16:39:12.578814 27343 heartbeater.cc:347] Connected to a master server at 127.25.71.126:44207
I20260928 16:39:12.579043 27343 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:12.579730 27343 heartbeater.cc:511] Master 127.25.71.126:44207 requested a full tablet report, sending...
I20260928 16:39:12.581004 27000 ts_manager.cc:194] Registered new tserver with Master: 762e1920adbd485d87832a7faa7ed2d0 (127.25.71.66:33913)
I20260928 16:39:12.581687 27000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:37901
WARNING: no policy specified for kudu/127.25.71.67@KRBTEST.COM; defaulting to no policy
I20260928 16:39:12.591176 27349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.582244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44207 (local address 127.25.71.66:50453)
0928 16:39:12.582690 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:12.582711 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:12.582892 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:12.583147 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:12.583150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:12.583416 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:12.583547 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.583554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.584372 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.584375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.584981 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.584990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.585476 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.585480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.585575 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.586270 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:12.586296 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:12.586870 (+ 574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:12.588945 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:12.588950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:12.588952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:12.589275 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:12.589730 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:12.589733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:12.589735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:12.589838 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:12.590441 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:12.590445 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:12.590564 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 16:39:12.590722 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 16:39:12.590951 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":160,"threads_started":1}
I20260928 16:39:12.591159 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.582276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:50453 (local address 127.25.71.126:44207)
0928 16:39:12.582401 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:12.582406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:12.582831 (+ 425us) server_negotiation.cc:407] Connection header received
0928 16:39:12.582990 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:12.582994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:12.583048 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:12.583117 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:12.583664 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.584237 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.585124 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.585363 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.585454 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.587194 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:12.587197 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:12.587199 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:12.587231 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:12.588769 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:12.589465 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:12.589469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:12.589470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:12.589572 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:12.589955 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:12.589957 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:12.589959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:12.590320 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:12.590427 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:12.590792 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 16:39:12.590949 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
I20260928 16:39:12.615113 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:44207
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:42879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:12.739264 27352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:12.739575 27352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:12.739622 27352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:12.744517 27352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:12.744658 27352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:12.744710 27352 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:39:12.744751 27352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:39:12.751902 27352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:42879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.71.126:44207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:12.753114 27352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:12.754289 27352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:12.766072 27360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:12.766052 27358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:12.766158 27357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:12.766469 27352 server_base.cc:1034] running on GCE node
I20260928 16:39:12.767174 27352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:12.768003 27352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:12.769214 27352 hybrid_clock.cc:648] HybridClock initialized: now 1790613552769191 us; error 42 us; skew 500 ppm
Sep 28 16:39:12 dist-test-slave-nqgh krb5kdc[26955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613552, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:12.772572 27352 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:39:12.773731 27352 webserver.cc:460] Webserver started at http://127.25.71.67:43285/ using document root <none> and password file <none>
I20260928 16:39:12.774386 27352 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:12.774463 27352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:12.774724 27352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:12.777072 27352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c"
format_stamp: "Formatted at 2026-09-28 16:39:12 on dist-test-slave-nqgh"
server_key: "9b08d2022b620e2e071db3d51868a1a2"
server_key_iv: "9db34b8294aed91ea014bab05cad5197"
server_key_version: "encryptionkey@0"
I20260928 16:39:12.777694 27352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c"
format_stamp: "Formatted at 2026-09-28 16:39:12 on dist-test-slave-nqgh"
server_key: "9b08d2022b620e2e071db3d51868a1a2"
server_key_iv: "9db34b8294aed91ea014bab05cad5197"
server_key_version: "encryptionkey@0"
I20260928 16:39:12.782450 27352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 16:39:12.785913 27367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:12.788131 27352 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:12.788327 27352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c"
format_stamp: "Formatted at 2026-09-28 16:39:12 on dist-test-slave-nqgh"
server_key: "9b08d2022b620e2e071db3d51868a1a2"
server_key_iv: "9db34b8294aed91ea014bab05cad5197"
server_key_version: "encryptionkey@0"
I20260928 16:39:12.788494 27352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:12.804512 27352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:12.807888 27352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:12.808094 27352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:12.809855 27352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:12.809918 27352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:12.809974 27352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:12.810004 27352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:12.821965 27352 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:35633
I20260928 16:39:12.822683 27482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:35633 every 8 connection(s)
I20260928 16:39:12.823184 27352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
Sep 28 16:39:12 dist-test-slave-nqgh krb5kdc[26955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613552, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:12.823882 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 27352
I20260928 16:39:12.824023 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:39:12.824357 25885 external_mini_cluster.cc:1428] Setting key b122f828014824042d3799ff32428b88
I20260928 16:39:12.830914 27377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.811884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44207 (local address 127.25.71.67:43789)
0928 16:39:12.812539 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:12.812606 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:12.814328 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:12.815202 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:12.815216 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:12.815841 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:12.816759 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.816791 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.818175 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.818180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.819841 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.819861 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.820728 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.820734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.821000 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.821913 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:12.821942 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:12.824525 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:12.826754 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:12.826761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:12.826781 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:12.827309 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:12.827713 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:12.827716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:12.827718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:12.828011 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:12.828565 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:12.828575 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:12.828966 (+ 391us) client_negotiation.cc:769] Sending connection context
0928 16:39:12.829249 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 16:39:12.829678 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":186,"threads_started":1}
I20260928 16:39:12.831305 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.811393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:43789 (local address 127.25.71.126:44207)
0928 16:39:12.811597 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:12.811604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:12.812773 (+ 1169us) server_negotiation.cc:407] Connection header received
0928 16:39:12.814661 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:12.814666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:12.814749 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:12.814838 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:12.817073 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.817972 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.820077 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.820492 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.820680 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.824781 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:12.824794 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:12.824797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:12.824836 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:12.826531 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:12.827462 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:12.827465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:12.827466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:12.827576 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:12.828186 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:12.828189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:12.828190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:12.828440 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:12.828539 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:12.830898 (+ 2359us) server_negotiation.cc:299] Negotiation successful
0928 16:39:12.831133 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:39:12.832909 27483 heartbeater.cc:347] Connected to a master server at 127.25.71.126:44207
I20260928 16:39:12.833133 27483 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:12.833830 27483 heartbeater.cc:511] Master 127.25.71.126:44207 requested a full tablet report, sending...
I20260928 16:39:12.835088 27000 ts_manager.cc:194] Registered new tserver with Master: e465f4d9b62a4cc98ff3770be6eb6f1c (127.25.71.67:35633)
I20260928 16:39:12.835708 27000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:43789
I20260928 16:39:12.839401 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:39:12.845628 27489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.835998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44207 (local address 127.25.71.67:49871)
0928 16:39:12.836446 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:12.836461 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:12.836572 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:12.836805 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:12.836808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:12.837068 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:12.837232 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.837241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.838164 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.838169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.838926 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.838940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.839504 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.839507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.839617 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.840380 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:12.840397 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:12.840924 (+ 527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:12.843313 (+ 2389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:12.843317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:12.843319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:12.843638 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:12.844148 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:12.844151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:12.844153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:12.844279 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:12.844881 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:12.844884 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:12.845034 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 16:39:12.845225 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 16:39:12.845424 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":159,"threads_started":1}
I20260928 16:39:12.845755 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.836057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:49871 (local address 127.25.71.126:44207)
0928 16:39:12.836189 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:12.836193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:12.836543 (+ 350us) server_negotiation.cc:407] Connection header received
0928 16:39:12.836661 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:12.836664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:12.836709 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:12.836772 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:12.837366 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.837995 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.839058 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.839356 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.839453 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.841182 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:12.841185 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:12.841187 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:12.841219 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:12.843108 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:12.843797 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:12.843801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:12.843803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:12.843962 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:12.844451 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:12.844455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:12.844457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:12.844732 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:12.844851 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:12.845344 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 16:39:12.845563 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 16:39:12 dist-test-slave-nqgh krb5kdc[26955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613552, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:39:12 dist-test-slave-nqgh krb5kdc[26955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613552, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:12.873430 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.861751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.25.71.126:44207)
0928 16:39:12.861911 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:12.861915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:12.861927 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:39:12.861973 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:12.861976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:12.862017 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:12.862200 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:12.863334 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.863892 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.864890 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.865117 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.865213 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.869062 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:12.869066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:12.869068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:12.869108 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:12.871617 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:12.872170 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:12.872173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:12.872175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:12.872259 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:12.872533 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:12.872536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:12.872537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:12.872742 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:12.872865 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:12.873127 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 16:39:12.873249 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:39:12.878162 27000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:39:12.882432 27000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:39:12.903028 27500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.895267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:33913 (local address 127.0.0.1:56952)
0928 16:39:12.896416 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:12.896495 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:12.896667 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:12.897633 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:12.897640 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:12.897663 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:12.897834 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.897841 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.899684 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.899689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.900825 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.900838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.902173 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.902177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.902336 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.902462 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 16:39:12.902591 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 16:39:12.902835 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"mutex_wait_us":73,"thread_start_us":112,"threads_started":1}
I20260928 16:39:12.903064 27504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.896265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56952 (local address 127.25.71.66:33913)
0928 16:39:12.897022 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:12.897045 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:12.897075 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:39:12.897170 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:12.897175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:12.897448 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:12.897637 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:12.897949 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.899290 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.900985 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.901981 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.902390 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.902646 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:12.902728 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 16:39:12.902886 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":120,"threads_started":1}
I20260928 16:39:12.903319 27503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.896264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54658 (local address 127.25.71.65:39791)
0928 16:39:12.896615 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:12.896624 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:12.897417 (+ 793us) server_negotiation.cc:407] Connection header received
0928 16:39:12.897518 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:12.897523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:12.897764 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:12.897948 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:12.898228 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.899398 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.901025 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.901861 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.902073 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.902194 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:12.902798 (+ 604us) server_negotiation.cc:299] Negotiation successful
0928 16:39:12.902931 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
I20260928 16:39:12.904008 27502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.895398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:35633 (local address 127.0.0.1:36764)
0928 16:39:12.897007 (+ 1609us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:12.897022 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:12.897125 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:12.897863 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:12.897867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:12.897888 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:12.898039 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.898046 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.899179 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.899183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.900797 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.900811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.902259 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.902263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.903695 (+ 1432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.903718 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:39:12.903793 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 16:39:12.903876 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1521,"mutex_wait_us":144,"thread_start_us":224,"threads_started":1}
I20260928 16:39:12.904146 27505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.896649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36764 (local address 127.25.71.67:35633)
0928 16:39:12.897308 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:12.897320 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:12.897342 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:39:12.897419 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:12.897423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:12.897601 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:12.897734 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:12.898183 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.899056 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.900966 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.901865 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.901976 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.902184 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:12.903869 (+ 1685us) server_negotiation.cc:299] Negotiation successful
0928 16:39:12.903998 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":125,"threads_started":1}
I20260928 16:39:12.904301 27501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.895625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:39791 (local address 127.0.0.1:54658)
0928 16:39:12.897305 (+ 1680us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:12.897315 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:12.897402 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:12.897980 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:12.897983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:12.898002 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:12.898099 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.898105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.899505 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.899508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.900861 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.900869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.902258 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.902263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.902455 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.902476 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 16:39:12.902596 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 16:39:12.904196 (+ 1600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1629,"mutex_wait_us":1554,"thread_start_us":125,"threads_started":1}
I20260928 16:39:12.907330 27277 tablet_service.cc:1467] Processing CreateTablet for tablet 37ed71bc75124c418da20d37d87b1ced (DEFAULT_TABLE table=test-table [id=3d44644d265b4ac093f4314da49ff0ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:12.907315 27137 tablet_service.cc:1467] Processing CreateTablet for tablet 37ed71bc75124c418da20d37d87b1ced (DEFAULT_TABLE table=test-table [id=3d44644d265b4ac093f4314da49ff0ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:12.908563 27137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ed71bc75124c418da20d37d87b1ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:12.908563 27277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ed71bc75124c418da20d37d87b1ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:12.908093 27417 tablet_service.cc:1467] Processing CreateTablet for tablet 37ed71bc75124c418da20d37d87b1ced (DEFAULT_TABLE table=test-table [id=3d44644d265b4ac093f4314da49ff0ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:12.909408 27417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ed71bc75124c418da20d37d87b1ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:12.919201 27507 tablet_bootstrap.cc:492] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd: Bootstrap starting.
I20260928 16:39:12.920894 27506 tablet_bootstrap.cc:492] T 37ed71bc75124c418da20d37d87b1ced P 762e1920adbd485d87832a7faa7ed2d0: Bootstrap starting.
I20260928 16:39:12.921363 27508 tablet_bootstrap.cc:492] T 37ed71bc75124c418da20d37d87b1ced P e465f4d9b62a4cc98ff3770be6eb6f1c: Bootstrap starting.
I20260928 16:39:12.922343 27507 tablet_bootstrap.cc:654] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:12.923596 27507 log.cc:824] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:12.923954 27506 tablet_bootstrap.cc:654] T 37ed71bc75124c418da20d37d87b1ced P 762e1920adbd485d87832a7faa7ed2d0: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:12.924233 27508 tablet_bootstrap.cc:654] T 37ed71bc75124c418da20d37d87b1ced P e465f4d9b62a4cc98ff3770be6eb6f1c: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:12.925029 27506 log.cc:824] T 37ed71bc75124c418da20d37d87b1ced P 762e1920adbd485d87832a7faa7ed2d0: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:12.925508 27508 log.cc:824] T 37ed71bc75124c418da20d37d87b1ced P e465f4d9b62a4cc98ff3770be6eb6f1c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:12.926905 27507 tablet_bootstrap.cc:492] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd: No bootstrap required, opened a new log
I20260928 16:39:12.927098 27507 ts_tablet_manager.cc:1397] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260928 16:39:12.927881 27506 tablet_bootstrap.cc:492] T 37ed71bc75124c418da20d37d87b1ced P 762e1920adbd485d87832a7faa7ed2d0: No bootstrap required, opened a new log
I20260928 16:39:12.928117 27508 tablet_bootstrap.cc:492] T 37ed71bc75124c418da20d37d87b1ced P e465f4d9b62a4cc98ff3770be6eb6f1c: No bootstrap required, opened a new log
I20260928 16:39:12.928139 27506 ts_tablet_manager.cc:1397] T 37ed71bc75124c418da20d37d87b1ced P 762e1920adbd485d87832a7faa7ed2d0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 16:39:12.928342 27508 ts_tablet_manager.cc:1397] T 37ed71bc75124c418da20d37d87b1ced P e465f4d9b62a4cc98ff3770be6eb6f1c: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260928 16:39:12.930943 27507 raft_consensus.cc:348] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } } peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } }
I20260928 16:39:12.931239 27507 raft_consensus.cc:374] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:12.931303 27507 raft_consensus.cc:729] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 822bbcfb122f43528736000f607ffdfd, State: Initialized, Role: FOLLOWER
I20260928 16:39:12.931851 27507 consensus_queue.cc:260] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } } peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } }
I20260928 16:39:12.933471 27507 ts_tablet_manager.cc:1428] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 16:39:12.933071 27506 raft_consensus.cc:348] T 37ed71bc75124c418da20d37d87b1ced P 762e1920adbd485d87832a7faa7ed2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } } peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } }
I20260928 16:39:12.933619 27506 raft_consensus.cc:374] T 37ed71bc75124c418da20d37d87b1ced P 762e1920adbd485d87832a7faa7ed2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:12.933699 27506 raft_consensus.cc:729] T 37ed71bc75124c418da20d37d87b1ced P 762e1920adbd485d87832a7faa7ed2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762e1920adbd485d87832a7faa7ed2d0, State: Initialized, Role: FOLLOWER
I20260928 16:39:12.933878 27203 heartbeater.cc:503] Master 127.25.71.126:44207 was elected leader, sending a full tablet report...
I20260928 16:39:12.934059 27508 raft_consensus.cc:348] T 37ed71bc75124c418da20d37d87b1ced P e465f4d9b62a4cc98ff3770be6eb6f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } } peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } }
I20260928 16:39:12.934371 27508 raft_consensus.cc:374] T 37ed71bc75124c418da20d37d87b1ced P e465f4d9b62a4cc98ff3770be6eb6f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:12.934432 27508 raft_consensus.cc:729] T 37ed71bc75124c418da20d37d87b1ced P e465f4d9b62a4cc98ff3770be6eb6f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e465f4d9b62a4cc98ff3770be6eb6f1c, State: Initialized, Role: FOLLOWER
I20260928 16:39:12.934422 27506 consensus_queue.cc:260] T 37ed71bc75124c418da20d37d87b1ced P 762e1920adbd485d87832a7faa7ed2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } } peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } }
I20260928 16:39:12.934985 27508 consensus_queue.cc:260] T 37ed71bc75124c418da20d37d87b1ced P e465f4d9b62a4cc98ff3770be6eb6f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } } peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } }
I20260928 16:39:12.935858 27506 ts_tablet_manager.cc:1428] T 37ed71bc75124c418da20d37d87b1ced P 762e1920adbd485d87832a7faa7ed2d0: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260928 16:39:12.936081 27508 ts_tablet_manager.cc:1428] T 37ed71bc75124c418da20d37d87b1ced P e465f4d9b62a4cc98ff3770be6eb6f1c: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260928 16:39:12.936323 27343 heartbeater.cc:503] Master 127.25.71.126:44207 was elected leader, sending a full tablet report...
I20260928 16:39:12.936483 27483 heartbeater.cc:503] Master 127.25.71.126:44207 was elected leader, sending a full tablet report...
I20260928 16:39:12.954248 27512 raft_consensus.cc:482] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:39:12.954618 27512 raft_consensus.cc:504] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } } peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } }
I20260928 16:39:12.956192 27512 leader_election.cc:288] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 762e1920adbd485d87832a7faa7ed2d0 (127.25.71.66:33913), e465f4d9b62a4cc98ff3770be6eb6f1c (127.25.71.67:35633)
I20260928 16:39:12.961376 27505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.957031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:49721 (local address 127.25.71.67:35633)
0928 16:39:12.957223 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:12.957229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:12.957257 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:39:12.957320 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:12.957322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:12.957363 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:12.957433 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:12.957756 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.958459 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.960118 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.960739 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.960870 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.960941 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:12.961127 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 16:39:12.961222 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 16:39:12.961403 27515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.956631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:35633 (local address 127.25.71.65:49721)
0928 16:39:12.957044 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:12.957062 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:12.957191 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:12.957468 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:12.957471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:12.957492 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:12.957623 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.957628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.958571 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.958575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.959934 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.959949 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.960874 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.960877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.960982 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.961010 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 16:39:12.961102 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 16:39:12.961207 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":1,"thread_start_us":176,"threads_started":1}
I20260928 16:39:12.961485 27504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.956838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:37175 (local address 127.25.71.66:33913)
0928 16:39:12.957114 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:12.957120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:12.957370 (+ 250us) server_negotiation.cc:407] Connection header received
0928 16:39:12.957543 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:12.957545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:12.957592 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:12.957657 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:12.958091 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.958817 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.960229 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.960807 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.960939 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.961001 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:12.961223 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 16:39:12.961333 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260928 16:39:12.962044 27516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:12.956700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:33913 (local address 127.25.71.65:37175)
0928 16:39:12.957258 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:12.957272 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:12.957413 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:12.957679 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:12.957682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:12.957828 (+ 146us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:12.957954 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.957960 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.958933 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.958936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.960062 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:12.960074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:12.960940 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:12.960944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:12.961072 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:12.961099 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:39:12.961726 (+ 627us) client_negotiation.cc:240] Negotiation successful
0928 16:39:12.961839 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":27,"thread_start_us":175,"threads_started":1}
I20260928 16:39:12.962456 27437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ed71bc75124c418da20d37d87b1ced" candidate_uuid: "822bbcfb122f43528736000f607ffdfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" is_pre_election: true
I20260928 16:39:12.962857 27297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ed71bc75124c418da20d37d87b1ced" candidate_uuid: "822bbcfb122f43528736000f607ffdfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762e1920adbd485d87832a7faa7ed2d0" is_pre_election: true
I20260928 16:39:12.962960 27437 raft_consensus.cc:2454] T 37ed71bc75124c418da20d37d87b1ced P e465f4d9b62a4cc98ff3770be6eb6f1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 822bbcfb122f43528736000f607ffdfd in term 0.
I20260928 16:39:12.963166 27297 raft_consensus.cc:2454] T 37ed71bc75124c418da20d37d87b1ced P 762e1920adbd485d87832a7faa7ed2d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 822bbcfb122f43528736000f607ffdfd in term 0.
I20260928 16:39:12.963724 27089 leader_election.cc:302] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 822bbcfb122f43528736000f607ffdfd, e465f4d9b62a4cc98ff3770be6eb6f1c; no voters:
I20260928 16:39:12.964066 27512 raft_consensus.cc:2781] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:39:12.964154 27512 raft_consensus.cc:482] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:39:12.964198 27512 raft_consensus.cc:3037] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:12.965574 27512 raft_consensus.cc:504] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } } peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } }
I20260928 16:39:12.966006 27512 leader_election.cc:288] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd [CANDIDATE]: Term 1 election: Requested vote from peers 762e1920adbd485d87832a7faa7ed2d0 (127.25.71.66:33913), e465f4d9b62a4cc98ff3770be6eb6f1c (127.25.71.67:35633)
I20260928 16:39:12.966492 27437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ed71bc75124c418da20d37d87b1ced" candidate_uuid: "822bbcfb122f43528736000f607ffdfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c"
I20260928 16:39:12.966532 27297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ed71bc75124c418da20d37d87b1ced" candidate_uuid: "822bbcfb122f43528736000f607ffdfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762e1920adbd485d87832a7faa7ed2d0"
I20260928 16:39:12.966693 27437 raft_consensus.cc:3037] T 37ed71bc75124c418da20d37d87b1ced P e465f4d9b62a4cc98ff3770be6eb6f1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:12.966782 27297 raft_consensus.cc:3037] T 37ed71bc75124c418da20d37d87b1ced P 762e1920adbd485d87832a7faa7ed2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:12.968091 27437 raft_consensus.cc:2454] T 37ed71bc75124c418da20d37d87b1ced P e465f4d9b62a4cc98ff3770be6eb6f1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 822bbcfb122f43528736000f607ffdfd in term 1.
I20260928 16:39:12.968501 27297 raft_consensus.cc:2454] T 37ed71bc75124c418da20d37d87b1ced P 762e1920adbd485d87832a7faa7ed2d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 822bbcfb122f43528736000f607ffdfd in term 1.
I20260928 16:39:12.968567 27089 leader_election.cc:302] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 822bbcfb122f43528736000f607ffdfd, e465f4d9b62a4cc98ff3770be6eb6f1c; no voters:
I20260928 16:39:12.968778 27512 raft_consensus.cc:2781] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:12.969062 27512 raft_consensus.cc:686] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd [term 1 LEADER]: Becoming Leader. State: Replica: 822bbcfb122f43528736000f607ffdfd, State: Running, Role: LEADER
I20260928 16:39:12.969514 27512 consensus_queue.cc:237] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } } peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } }
I20260928 16:39:12.974092 27000 catalog_manager.cc:5495] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 822bbcfb122f43528736000f607ffdfd (127.25.71.65). New cstate: current_term: 1 leader_uuid: "822bbcfb122f43528736000f607ffdfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } health_report { overall_health: HEALTHY } } }
I20260928 16:39:13.008275 27503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:13.003494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54662 (local address 127.25.71.65:39791)
0928 16:39:13.003687 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:13.003692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:13.003713 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:39:13.003973 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:13.003977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:13.004050 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:13.004161 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:13.004542 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.005164 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.006237 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.006480 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.006575 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:13.007224 (+ 649us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:13.007721 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:13.007819 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:13.008040 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 16:39:13.008113 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 16:39:13.016563 27437 raft_consensus.cc:1261] T 37ed71bc75124c418da20d37d87b1ced P e465f4d9b62a4cc98ff3770be6eb6f1c [term 1 FOLLOWER]: Refusing update from remote peer 822bbcfb122f43528736000f607ffdfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:13.016664 27297 raft_consensus.cc:1261] T 37ed71bc75124c418da20d37d87b1ced P 762e1920adbd485d87832a7faa7ed2d0 [term 1 FOLLOWER]: Refusing update from remote peer 822bbcfb122f43528736000f607ffdfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:13.017549 27517 consensus_queue.cc:1035] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:13.017879 27512 consensus_queue.cc:1035] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:13.026542 27520 mvcc.cc:205] Tried to move back new op lower bound from 7334353113144631296 to 7334353112966594560. Current Snapshot: MvccSnapshot[applied={T|T < 7334353113144631296}]
I20260928 16:39:13.044906 27000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:13.045217 27000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:13.048969 27000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9: Sending DeleteTablet for 3 replicas of tablet 37ed71bc75124c418da20d37d87b1ced
I20260928 16:39:13.049971 27137 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ed71bc75124c418da20d37d87b1ced with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:13 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:54658
I20260928 16:39:13.049960 27277 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ed71bc75124c418da20d37d87b1ced with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:13 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:56952
I20260928 16:39:13.050233 27417 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ed71bc75124c418da20d37d87b1ced with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:13 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:36764
I20260928 16:39:13.050825 27530 tablet_replica.cc:330] T 37ed71bc75124c418da20d37d87b1ced P 762e1920adbd485d87832a7faa7ed2d0: stopping tablet replica
I20260928 16:39:13.051100 27531 tablet_replica.cc:330] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd: stopping tablet replica
I20260928 16:39:13.051433 27531 raft_consensus.cc:2229] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:39:13.051808 27530 raft_consensus.cc:2229] T 37ed71bc75124c418da20d37d87b1ced P 762e1920adbd485d87832a7faa7ed2d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:13.051853 27531 raft_consensus.cc:2258] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:13.052121 27532 tablet_replica.cc:330] T 37ed71bc75124c418da20d37d87b1ced P e465f4d9b62a4cc98ff3770be6eb6f1c: stopping tablet replica
I20260928 16:39:13.052300 27530 raft_consensus.cc:2258] T 37ed71bc75124c418da20d37d87b1ced P 762e1920adbd485d87832a7faa7ed2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:13.052438 27532 raft_consensus.cc:2229] T 37ed71bc75124c418da20d37d87b1ced P e465f4d9b62a4cc98ff3770be6eb6f1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:13.052394 27000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 16:39:13.052743 27532 raft_consensus.cc:2258] T 37ed71bc75124c418da20d37d87b1ced P e465f4d9b62a4cc98ff3770be6eb6f1c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 16:39:13.053079 27000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:39:13.054361 27530 ts_tablet_manager.cc:1905] T 37ed71bc75124c418da20d37d87b1ced P 762e1920adbd485d87832a7faa7ed2d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:13.055495 27531 ts_tablet_manager.cc:1905] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:13.055953 27532 ts_tablet_manager.cc:1905] T 37ed71bc75124c418da20d37d87b1ced P e465f4d9b62a4cc98ff3770be6eb6f1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:13.059278 27531 ts_tablet_manager.cc:1918] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:39:13.059412 27531 log.cc:1196] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-0/wal/wals/37ed71bc75124c418da20d37d87b1ced
I20260928 16:39:13.059748 27531 ts_tablet_manager.cc:1939] T 37ed71bc75124c418da20d37d87b1ced P 822bbcfb122f43528736000f607ffdfd: Deleting consensus metadata
I20260928 16:39:13.060685 27277 tablet_service.cc:1467] Processing CreateTablet for tablet fe9fff0a73f94306a3da314211151091 (DEFAULT_TABLE table=test-table [id=09919e3e0a604b0cb0563a1911cd1495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:13.060667 27417 tablet_service.cc:1467] Processing CreateTablet for tablet fe9fff0a73f94306a3da314211151091 (DEFAULT_TABLE table=test-table [id=09919e3e0a604b0cb0563a1911cd1495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:13.060983 27277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe9fff0a73f94306a3da314211151091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:13.061113 27417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe9fff0a73f94306a3da314211151091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:13.063773 27530 ts_tablet_manager.cc:1918] T 37ed71bc75124c418da20d37d87b1ced P 762e1920adbd485d87832a7faa7ed2d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:39:13.063738 27137 tablet_service.cc:1467] Processing CreateTablet for tablet fe9fff0a73f94306a3da314211151091 (DEFAULT_TABLE table=test-table [id=09919e3e0a604b0cb0563a1911cd1495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:13.063933 27530 log.cc:1196] T 37ed71bc75124c418da20d37d87b1ced P 762e1920adbd485d87832a7faa7ed2d0: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/37ed71bc75124c418da20d37d87b1ced
I20260928 16:39:13.064033 27137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe9fff0a73f94306a3da314211151091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:13.064292 27530 ts_tablet_manager.cc:1939] T 37ed71bc75124c418da20d37d87b1ced P 762e1920adbd485d87832a7faa7ed2d0: Deleting consensus metadata
I20260928 16:39:13.065963 27508 tablet_bootstrap.cc:492] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c: Bootstrap starting.
I20260928 16:39:13.066272 27532 ts_tablet_manager.cc:1918] T 37ed71bc75124c418da20d37d87b1ced P e465f4d9b62a4cc98ff3770be6eb6f1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:39:13.066428 27532 log.cc:1196] T 37ed71bc75124c418da20d37d87b1ced P e465f4d9b62a4cc98ff3770be6eb6f1c: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/37ed71bc75124c418da20d37d87b1ced
I20260928 16:39:13.066746 27507 tablet_bootstrap.cc:492] T fe9fff0a73f94306a3da314211151091 P 822bbcfb122f43528736000f607ffdfd: Bootstrap starting.
I20260928 16:39:13.066859 27532 ts_tablet_manager.cc:1939] T 37ed71bc75124c418da20d37d87b1ced P e465f4d9b62a4cc98ff3770be6eb6f1c: Deleting consensus metadata
I20260928 16:39:13.067173 27508 tablet_bootstrap.cc:654] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:13.067718 27507 tablet_bootstrap.cc:654] T fe9fff0a73f94306a3da314211151091 P 822bbcfb122f43528736000f607ffdfd: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:13.068629 26987 catalog_manager.cc:4841] TS 822bbcfb122f43528736000f607ffdfd (127.25.71.65:39791): tablet 37ed71bc75124c418da20d37d87b1ced (table test-table [id=3d44644d265b4ac093f4314da49ff0ac]) successfully deleted
I20260928 16:39:13.068629 26986 catalog_manager.cc:4841] TS 762e1920adbd485d87832a7faa7ed2d0 (127.25.71.66:33913): tablet 37ed71bc75124c418da20d37d87b1ced (table test-table [id=3d44644d265b4ac093f4314da49ff0ac]) successfully deleted
I20260928 16:39:13.069433 27506 tablet_bootstrap.cc:492] T fe9fff0a73f94306a3da314211151091 P 762e1920adbd485d87832a7faa7ed2d0: Bootstrap starting.
I20260928 16:39:13.069636 27508 tablet_bootstrap.cc:492] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c: No bootstrap required, opened a new log
I20260928 16:39:13.069746 27508 ts_tablet_manager.cc:1397] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 16:39:13.070062 26985 catalog_manager.cc:4841] TS e465f4d9b62a4cc98ff3770be6eb6f1c (127.25.71.67:35633): tablet 37ed71bc75124c418da20d37d87b1ced (table test-table [id=3d44644d265b4ac093f4314da49ff0ac]) successfully deleted
I20260928 16:39:13.070287 27508 raft_consensus.cc:348] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } }
I20260928 16:39:13.070429 27508 raft_consensus.cc:374] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:13.070462 27508 raft_consensus.cc:729] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e465f4d9b62a4cc98ff3770be6eb6f1c, State: Initialized, Role: FOLLOWER
I20260928 16:39:13.070554 27506 tablet_bootstrap.cc:654] T fe9fff0a73f94306a3da314211151091 P 762e1920adbd485d87832a7faa7ed2d0: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:13.070744 27508 consensus_queue.cc:260] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } }
I20260928 16:39:13.071794 27508 ts_tablet_manager.cc:1428] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:39:13.072641 27507 tablet_bootstrap.cc:492] T fe9fff0a73f94306a3da314211151091 P 822bbcfb122f43528736000f607ffdfd: No bootstrap required, opened a new log
I20260928 16:39:13.072793 27507 ts_tablet_manager.cc:1397] T fe9fff0a73f94306a3da314211151091 P 822bbcfb122f43528736000f607ffdfd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 16:39:13.072821 27506 tablet_bootstrap.cc:492] T fe9fff0a73f94306a3da314211151091 P 762e1920adbd485d87832a7faa7ed2d0: No bootstrap required, opened a new log
I20260928 16:39:13.072914 27506 ts_tablet_manager.cc:1397] T fe9fff0a73f94306a3da314211151091 P 762e1920adbd485d87832a7faa7ed2d0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 16:39:13.073421 27506 raft_consensus.cc:348] T fe9fff0a73f94306a3da314211151091 P 762e1920adbd485d87832a7faa7ed2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } }
I20260928 16:39:13.073557 27506 raft_consensus.cc:374] T fe9fff0a73f94306a3da314211151091 P 762e1920adbd485d87832a7faa7ed2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:13.073590 27506 raft_consensus.cc:729] T fe9fff0a73f94306a3da314211151091 P 762e1920adbd485d87832a7faa7ed2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762e1920adbd485d87832a7faa7ed2d0, State: Initialized, Role: FOLLOWER
I20260928 16:39:13.073720 27506 consensus_queue.cc:260] T fe9fff0a73f94306a3da314211151091 P 762e1920adbd485d87832a7faa7ed2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } }
I20260928 16:39:13.073676 27507 raft_consensus.cc:348] T fe9fff0a73f94306a3da314211151091 P 822bbcfb122f43528736000f607ffdfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } }
I20260928 16:39:13.073861 27507 raft_consensus.cc:374] T fe9fff0a73f94306a3da314211151091 P 822bbcfb122f43528736000f607ffdfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:13.073897 27507 raft_consensus.cc:729] T fe9fff0a73f94306a3da314211151091 P 822bbcfb122f43528736000f607ffdfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 822bbcfb122f43528736000f607ffdfd, State: Initialized, Role: FOLLOWER
I20260928 16:39:13.073992 27506 ts_tablet_manager.cc:1428] T fe9fff0a73f94306a3da314211151091 P 762e1920adbd485d87832a7faa7ed2d0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 16:39:13.074054 27507 consensus_queue.cc:260] T fe9fff0a73f94306a3da314211151091 P 822bbcfb122f43528736000f607ffdfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } }
I20260928 16:39:13.074365 27507 ts_tablet_manager.cc:1428] T fe9fff0a73f94306a3da314211151091 P 822bbcfb122f43528736000f607ffdfd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260928 16:39:13.074816 27344 tablet.cc:2369] T fe9fff0a73f94306a3da314211151091 P 762e1920adbd485d87832a7faa7ed2d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:39:13.075519 27484 tablet.cc:2369] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:39:13.232918 27514 raft_consensus.cc:482] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:39:13.233112 27514 raft_consensus.cc:504] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } }
I20260928 16:39:13.234424 27514 leader_election.cc:288] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 822bbcfb122f43528736000f607ffdfd (127.25.71.65:39791), 762e1920adbd485d87832a7faa7ed2d0 (127.25.71.66:33913)
I20260928 16:39:13.239669 27377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:13.234874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:33913 (local address 127.25.71.67:59629)
0928 16:39:13.235021 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:13.235039 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:13.235166 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:13.235519 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:13.235521 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:13.235539 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:13.235727 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:13.235735 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.236663 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.236667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:13.237752 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:13.237764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.239047 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.239052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:13.239192 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:13.239227 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:39:13.239357 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 16:39:13.239476 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":1,"spinlock_wait_cycles":18816}
I20260928 16:39:13.239827 27504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:13.235003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:59629 (local address 127.25.71.66:33913)
0928 16:39:13.235132 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:13.235137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:13.235157 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:39:13.235313 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:13.235316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:13.235375 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:13.235505 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:13.235896 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.236546 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.237943 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.238847 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.238980 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:13.239043 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:13.239402 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 16:39:13.239510 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 16:39:13.240406 27503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:13.234994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:46257 (local address 127.25.71.65:39791)
0928 16:39:13.235133 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:13.235137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:13.235594 (+ 457us) server_negotiation.cc:407] Connection header received
0928 16:39:13.235771 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:13.235775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:13.235845 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:13.235921 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:13.236316 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.237045 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.238907 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.239622 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.239767 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:13.239838 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:13.240182 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 16:39:13.240284 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:39:13.240440 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:13.234874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:39791 (local address 127.25.71.67:46257)
0928 16:39:13.235457 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:13.235476 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:13.235614 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:13.235978 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:13.235982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:13.236005 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:13.236163 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:13.236171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.237218 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.237221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:13.238453 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:13.238469 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.239869 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.239874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:13.239997 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:13.240027 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 16:39:13.240140 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 16:39:13.240256 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":170,"thread_start_us":126,"threads_started":1}
I20260928 16:39:13.240690 27297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9fff0a73f94306a3da314211151091" candidate_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762e1920adbd485d87832a7faa7ed2d0" is_pre_election: true
I20260928 16:39:13.240860 27297 raft_consensus.cc:2454] T fe9fff0a73f94306a3da314211151091 P 762e1920adbd485d87832a7faa7ed2d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e465f4d9b62a4cc98ff3770be6eb6f1c in term 0.
I20260928 16:39:13.241487 27370 leader_election.cc:302] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 762e1920adbd485d87832a7faa7ed2d0, e465f4d9b62a4cc98ff3770be6eb6f1c; no voters:
I20260928 16:39:13.241654 27157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9fff0a73f94306a3da314211151091" candidate_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822bbcfb122f43528736000f607ffdfd" is_pre_election: true
I20260928 16:39:13.241856 27514 raft_consensus.cc:2781] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:39:13.241916 27514 raft_consensus.cc:482] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:39:13.241967 27514 raft_consensus.cc:3037] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:13.242065 27157 raft_consensus.cc:2454] T fe9fff0a73f94306a3da314211151091 P 822bbcfb122f43528736000f607ffdfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e465f4d9b62a4cc98ff3770be6eb6f1c in term 0.
I20260928 16:39:13.243439 27514 raft_consensus.cc:504] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } }
I20260928 16:39:13.243928 27514 leader_election.cc:288] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c [CANDIDATE]: Term 1 election: Requested vote from peers 822bbcfb122f43528736000f607ffdfd (127.25.71.65:39791), 762e1920adbd485d87832a7faa7ed2d0 (127.25.71.66:33913)
I20260928 16:39:13.244532 27297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9fff0a73f94306a3da314211151091" candidate_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762e1920adbd485d87832a7faa7ed2d0"
I20260928 16:39:13.244541 27157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9fff0a73f94306a3da314211151091" candidate_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822bbcfb122f43528736000f607ffdfd"
I20260928 16:39:13.244699 27157 raft_consensus.cc:3037] T fe9fff0a73f94306a3da314211151091 P 822bbcfb122f43528736000f607ffdfd [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:13.244694 27297 raft_consensus.cc:3037] T fe9fff0a73f94306a3da314211151091 P 762e1920adbd485d87832a7faa7ed2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:13.245837 27157 raft_consensus.cc:2454] T fe9fff0a73f94306a3da314211151091 P 822bbcfb122f43528736000f607ffdfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e465f4d9b62a4cc98ff3770be6eb6f1c in term 1.
I20260928 16:39:13.245898 27297 raft_consensus.cc:2454] T fe9fff0a73f94306a3da314211151091 P 762e1920adbd485d87832a7faa7ed2d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e465f4d9b62a4cc98ff3770be6eb6f1c in term 1.
I20260928 16:39:13.246306 27371 leader_election.cc:302] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 822bbcfb122f43528736000f607ffdfd, e465f4d9b62a4cc98ff3770be6eb6f1c; no voters:
I20260928 16:39:13.246551 27514 raft_consensus.cc:2781] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:13.246814 27514 raft_consensus.cc:686] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c [term 1 LEADER]: Becoming Leader. State: Replica: e465f4d9b62a4cc98ff3770be6eb6f1c, State: Running, Role: LEADER
I20260928 16:39:13.247144 27514 consensus_queue.cc:237] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } }
I20260928 16:39:13.249958 26999 catalog_manager.cc:5495] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c reported cstate change: term changed from 0 to 1, leader changed from <none> to e465f4d9b62a4cc98ff3770be6eb6f1c (127.25.71.67). New cstate: current_term: 1 leader_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } health_report { overall_health: HEALTHY } } }
W20260928 16:39:13.290096 27204 tablet.cc:2369] T fe9fff0a73f94306a3da314211151091 P 822bbcfb122f43528736000f607ffdfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 16:39:13 dist-test-slave-nqgh krb5kdc[26955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613551, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:13.313390 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:13.301268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37776 (local address 127.25.71.126:44207)
0928 16:39:13.301450 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:13.301454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:13.301625 (+ 171us) server_negotiation.cc:407] Connection header received
0928 16:39:13.301806 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:13.301810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:13.301876 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:13.302041 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:13.303306 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.303942 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.305011 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.305260 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.305372 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:13.308758 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:13.308763 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:13.308765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:13.308805 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:13.310827 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:13.311700 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:13.311707 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:13.311708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:13.311809 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:13.312297 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:13.312300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:13.312301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:13.312547 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:13.312653 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:13.313089 (+ 436us) server_negotiation.cc:299] Negotiation successful
0928 16:39:13.313249 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 16:39:13.313666 27544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:13.301140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44207 (local address 127.0.0.1:37776)
0928 16:39:13.301506 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:13.301525 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:13.301658 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:13.302279 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:13.302286 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:13.302921 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:13.303078 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:13.303087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.304104 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.304108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:13.304832 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:13.304844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.305402 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.305405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:13.305478 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:13.306290 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:13.306304 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:13.308419 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:13.311021 (+ 2602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:13.311045 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:13.311132 (+ 87us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:13.311532 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:13.311955 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:13.311958 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:13.311959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:13.312158 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:13.312696 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:13.312704 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:13.312883 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 16:39:13.313005 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 16:39:13.313412 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":81,"thread_start_us":127,"threads_started":1}
I20260928 16:39:13.320056 26999 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:37776:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 16:39:13.321241 26999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:39:13.328480 27417 tablet_service.cc:1467] Processing CreateTablet for tablet f3a105cdecfd409f87f06861ed29f572 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=864cc5c53af94960b7244a308461c131]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:39:13.328480 27277 tablet_service.cc:1467] Processing CreateTablet for tablet f3a105cdecfd409f87f06861ed29f572 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=864cc5c53af94960b7244a308461c131]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:39:13.328497 27137 tablet_service.cc:1467] Processing CreateTablet for tablet f3a105cdecfd409f87f06861ed29f572 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=864cc5c53af94960b7244a308461c131]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:39:13.328965 27417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3a105cdecfd409f87f06861ed29f572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:13.328965 27277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3a105cdecfd409f87f06861ed29f572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:13.329090 27137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3a105cdecfd409f87f06861ed29f572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:13.332530 27507 tablet_bootstrap.cc:492] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd: Bootstrap starting.
I20260928 16:39:13.332610 27508 tablet_bootstrap.cc:492] T f3a105cdecfd409f87f06861ed29f572 P e465f4d9b62a4cc98ff3770be6eb6f1c: Bootstrap starting.
I20260928 16:39:13.333523 27508 tablet_bootstrap.cc:654] T f3a105cdecfd409f87f06861ed29f572 P e465f4d9b62a4cc98ff3770be6eb6f1c: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:13.333696 27507 tablet_bootstrap.cc:654] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:13.334846 27506 tablet_bootstrap.cc:492] T f3a105cdecfd409f87f06861ed29f572 P 762e1920adbd485d87832a7faa7ed2d0: Bootstrap starting.
I20260928 16:39:13.335163 27507 tablet_bootstrap.cc:492] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd: No bootstrap required, opened a new log
I20260928 16:39:13.335243 27507 ts_tablet_manager.cc:1397] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 16:39:13.335704 27507 raft_consensus.cc:348] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } }
I20260928 16:39:13.335845 27507 raft_consensus.cc:374] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:13.335880 27507 raft_consensus.cc:729] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 822bbcfb122f43528736000f607ffdfd, State: Initialized, Role: FOLLOWER
I20260928 16:39:13.335889 27506 tablet_bootstrap.cc:654] T f3a105cdecfd409f87f06861ed29f572 P 762e1920adbd485d87832a7faa7ed2d0: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:13.336076 27507 consensus_queue.cc:260] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } }
I20260928 16:39:13.336194 27508 tablet_bootstrap.cc:492] T f3a105cdecfd409f87f06861ed29f572 P e465f4d9b62a4cc98ff3770be6eb6f1c: No bootstrap required, opened a new log
I20260928 16:39:13.336300 27508 ts_tablet_manager.cc:1397] T f3a105cdecfd409f87f06861ed29f572 P e465f4d9b62a4cc98ff3770be6eb6f1c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 16:39:13.336382 27526 tablet_replica.cc:439] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } } }
I20260928 16:39:13.336544 27526 tablet_replica.cc:442] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:39:13.336807 27507 ts_tablet_manager.cc:1428] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 16:39:13.336867 27508 raft_consensus.cc:348] T f3a105cdecfd409f87f06861ed29f572 P e465f4d9b62a4cc98ff3770be6eb6f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } }
I20260928 16:39:13.337035 27508 raft_consensus.cc:374] T f3a105cdecfd409f87f06861ed29f572 P e465f4d9b62a4cc98ff3770be6eb6f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:13.337077 27508 raft_consensus.cc:729] T f3a105cdecfd409f87f06861ed29f572 P e465f4d9b62a4cc98ff3770be6eb6f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e465f4d9b62a4cc98ff3770be6eb6f1c, State: Initialized, Role: FOLLOWER
I20260928 16:39:13.337155 27506 tablet_bootstrap.cc:492] T f3a105cdecfd409f87f06861ed29f572 P 762e1920adbd485d87832a7faa7ed2d0: No bootstrap required, opened a new log
I20260928 16:39:13.337241 27506 ts_tablet_manager.cc:1397] T f3a105cdecfd409f87f06861ed29f572 P 762e1920adbd485d87832a7faa7ed2d0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:39:13.337216 27508 consensus_queue.cc:260] T f3a105cdecfd409f87f06861ed29f572 P e465f4d9b62a4cc98ff3770be6eb6f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } }
I20260928 16:39:13.337533 27508 ts_tablet_manager.cc:1428] T f3a105cdecfd409f87f06861ed29f572 P e465f4d9b62a4cc98ff3770be6eb6f1c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 16:39:13.337723 27506 raft_consensus.cc:348] T f3a105cdecfd409f87f06861ed29f572 P 762e1920adbd485d87832a7faa7ed2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } }
I20260928 16:39:13.337877 27506 raft_consensus.cc:374] T f3a105cdecfd409f87f06861ed29f572 P 762e1920adbd485d87832a7faa7ed2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:13.337915 27506 raft_consensus.cc:729] T f3a105cdecfd409f87f06861ed29f572 P 762e1920adbd485d87832a7faa7ed2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762e1920adbd485d87832a7faa7ed2d0, State: Initialized, Role: FOLLOWER
I20260928 16:39:13.338056 27506 consensus_queue.cc:260] T f3a105cdecfd409f87f06861ed29f572 P 762e1920adbd485d87832a7faa7ed2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } }
I20260928 16:39:13.337463 27514 tablet_replica.cc:439] T f3a105cdecfd409f87f06861ed29f572 P e465f4d9b62a4cc98ff3770be6eb6f1c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } } }
I20260928 16:39:13.338253 27514 tablet_replica.cc:442] T f3a105cdecfd409f87f06861ed29f572 P e465f4d9b62a4cc98ff3770be6eb6f1c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:39:13.338349 27506 ts_tablet_manager.cc:1428] T f3a105cdecfd409f87f06861ed29f572 P 762e1920adbd485d87832a7faa7ed2d0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 16:39:13.338405 27513 tablet_replica.cc:439] T f3a105cdecfd409f87f06861ed29f572 P 762e1920adbd485d87832a7faa7ed2d0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } } }
I20260928 16:39:13.338586 27513 tablet_replica.cc:442] T f3a105cdecfd409f87f06861ed29f572 P 762e1920adbd485d87832a7faa7ed2d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:39:13.398113 27526 raft_consensus.cc:482] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:39:13.398376 27526 raft_consensus.cc:504] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } }
I20260928 16:39:13.399143 27526 leader_election.cc:288] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 762e1920adbd485d87832a7faa7ed2d0 (127.25.71.66:33913), e465f4d9b62a4cc98ff3770be6eb6f1c (127.25.71.67:35633)
I20260928 16:39:13.399863 27297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a105cdecfd409f87f06861ed29f572" candidate_uuid: "822bbcfb122f43528736000f607ffdfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762e1920adbd485d87832a7faa7ed2d0" is_pre_election: true
I20260928 16:39:13.400043 27437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a105cdecfd409f87f06861ed29f572" candidate_uuid: "822bbcfb122f43528736000f607ffdfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" is_pre_election: true
I20260928 16:39:13.400202 27297 raft_consensus.cc:2454] T f3a105cdecfd409f87f06861ed29f572 P 762e1920adbd485d87832a7faa7ed2d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 822bbcfb122f43528736000f607ffdfd in term 0.
I20260928 16:39:13.400256 27437 raft_consensus.cc:2454] T f3a105cdecfd409f87f06861ed29f572 P e465f4d9b62a4cc98ff3770be6eb6f1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 822bbcfb122f43528736000f607ffdfd in term 0.
I20260928 16:39:13.400581 27090 leader_election.cc:302] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 762e1920adbd485d87832a7faa7ed2d0, 822bbcfb122f43528736000f607ffdfd; no voters:
I20260928 16:39:13.400827 27526 raft_consensus.cc:2781] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:39:13.400918 27526 raft_consensus.cc:482] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:39:13.400951 27526 raft_consensus.cc:3037] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:13.402490 27526 raft_consensus.cc:504] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } }
I20260928 16:39:13.403105 27526 leader_election.cc:288] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd [CANDIDATE]: Term 1 election: Requested vote from peers 762e1920adbd485d87832a7faa7ed2d0 (127.25.71.66:33913), e465f4d9b62a4cc98ff3770be6eb6f1c (127.25.71.67:35633)
I20260928 16:39:13.403680 27437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a105cdecfd409f87f06861ed29f572" candidate_uuid: "822bbcfb122f43528736000f607ffdfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c"
I20260928 16:39:13.403695 27297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a105cdecfd409f87f06861ed29f572" candidate_uuid: "822bbcfb122f43528736000f607ffdfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762e1920adbd485d87832a7faa7ed2d0"
I20260928 16:39:13.403893 27437 raft_consensus.cc:3037] T f3a105cdecfd409f87f06861ed29f572 P e465f4d9b62a4cc98ff3770be6eb6f1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:13.403898 27297 raft_consensus.cc:3037] T f3a105cdecfd409f87f06861ed29f572 P 762e1920adbd485d87832a7faa7ed2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:13.405051 27297 raft_consensus.cc:2454] T f3a105cdecfd409f87f06861ed29f572 P 762e1920adbd485d87832a7faa7ed2d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 822bbcfb122f43528736000f607ffdfd in term 1.
I20260928 16:39:13.405263 27437 raft_consensus.cc:2454] T f3a105cdecfd409f87f06861ed29f572 P e465f4d9b62a4cc98ff3770be6eb6f1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 822bbcfb122f43528736000f607ffdfd in term 1.
I20260928 16:39:13.405422 27090 leader_election.cc:302] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 762e1920adbd485d87832a7faa7ed2d0, 822bbcfb122f43528736000f607ffdfd; no voters:
I20260928 16:39:13.405642 27526 raft_consensus.cc:2781] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:13.405740 27526 raft_consensus.cc:686] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd [term 1 LEADER]: Becoming Leader. State: Replica: 822bbcfb122f43528736000f607ffdfd, State: Running, Role: LEADER
I20260928 16:39:13.405911 27526 consensus_queue.cc:237] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } }
I20260928 16:39:13.406761 27512 tablet_replica.cc:439] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd: TxnStatusTablet state changed. Reason: New leader 822bbcfb122f43528736000f607ffdfd. Latest consensus state: current_term: 1 leader_uuid: "822bbcfb122f43528736000f607ffdfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } } }
I20260928 16:39:13.406922 27512 tablet_replica.cc:442] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:39:13.407430 27549 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 16:39:13.407624 27549 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 16:39:13.408263 26998 catalog_manager.cc:5495] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 822bbcfb122f43528736000f607ffdfd (127.25.71.65). New cstate: current_term: 1 leader_uuid: "822bbcfb122f43528736000f607ffdfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } health_report { overall_health: UNKNOWN } } }
I20260928 16:39:13.431733 27544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:13.427851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:39791 (local address 127.0.0.1:54676)
0928 16:39:13.428057 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:13.428077 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:13.428232 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:13.428596 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:13.428600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:13.428614 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:39:13.428846 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:13.428855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.429759 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.429763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:13.430435 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:13.430445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.430971 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.430974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:13.431041 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:13.431099 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:39:13.431412 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:39:13.431429 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 16:39:13.431486 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 16:39:13.431613 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 16:39:13.431732 27503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:13.427998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.25.71.65:39791)
0928 16:39:13.428152 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:13.428157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:13.428172 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:39:13.428366 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:13.428368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:13.428444 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:13.428521 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:13.428994 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.429618 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.430560 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.430793 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.430881 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:13.431238 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:13.431329 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:13.431399 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:13.431507 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 16:39:13.431572 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:39:13.435778 27297 raft_consensus.cc:1261] T f3a105cdecfd409f87f06861ed29f572 P 762e1920adbd485d87832a7faa7ed2d0 [term 1 FOLLOWER]: Refusing update from remote peer 822bbcfb122f43528736000f607ffdfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:13.435981 27437 raft_consensus.cc:1261] T f3a105cdecfd409f87f06861ed29f572 P e465f4d9b62a4cc98ff3770be6eb6f1c [term 1 FOLLOWER]: Refusing update from remote peer 822bbcfb122f43528736000f607ffdfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:13.436245 27512 consensus_queue.cc:1035] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:13.437745 27512 consensus_queue.cc:1035] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:13.439042 27513 tablet_replica.cc:439] T f3a105cdecfd409f87f06861ed29f572 P 762e1920adbd485d87832a7faa7ed2d0: TxnStatusTablet state changed. Reason: New leader 822bbcfb122f43528736000f607ffdfd. Latest consensus state: current_term: 1 leader_uuid: "822bbcfb122f43528736000f607ffdfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } } }
I20260928 16:39:13.439173 27513 tablet_replica.cc:442] T f3a105cdecfd409f87f06861ed29f572 P 762e1920adbd485d87832a7faa7ed2d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:39:13.439750 27526 tablet_replica.cc:439] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "822bbcfb122f43528736000f607ffdfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } } }
I20260928 16:39:13.439872 27526 tablet_replica.cc:442] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:39:13.440599 27521 mvcc.cc:205] Tried to move back new op lower bound from 7334353114867507200 to 7334353114752122880. Current Snapshot: MvccSnapshot[applied={T|T < 7334353114867507200}]
I20260928 16:39:13.441890 27513 tablet_replica.cc:439] T f3a105cdecfd409f87f06861ed29f572 P 762e1920adbd485d87832a7faa7ed2d0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "822bbcfb122f43528736000f607ffdfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } } }
I20260928 16:39:13.442003 27513 tablet_replica.cc:442] T f3a105cdecfd409f87f06861ed29f572 P 762e1920adbd485d87832a7faa7ed2d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:39:13.445595 27514 tablet_replica.cc:439] T f3a105cdecfd409f87f06861ed29f572 P e465f4d9b62a4cc98ff3770be6eb6f1c: TxnStatusTablet state changed. Reason: New leader 822bbcfb122f43528736000f607ffdfd. Latest consensus state: current_term: 1 leader_uuid: "822bbcfb122f43528736000f607ffdfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } } }
I20260928 16:39:13.445814 27514 tablet_replica.cc:442] T f3a105cdecfd409f87f06861ed29f572 P e465f4d9b62a4cc98ff3770be6eb6f1c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:39:13.446344 27526 tablet_replica.cc:439] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "822bbcfb122f43528736000f607ffdfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } } }
I20260928 16:39:13.446504 27526 tablet_replica.cc:442] T f3a105cdecfd409f87f06861ed29f572 P 822bbcfb122f43528736000f607ffdfd: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:39:13.448257 27514 tablet_replica.cc:439] T f3a105cdecfd409f87f06861ed29f572 P e465f4d9b62a4cc98ff3770be6eb6f1c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "822bbcfb122f43528736000f607ffdfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 } } peers { permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 } } peers { permanent_uuid: "e465f4d9b62a4cc98ff3770be6eb6f1c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35633 } } }
I20260928 16:39:13.448454 27514 tablet_replica.cc:442] T f3a105cdecfd409f87f06861ed29f572 P e465f4d9b62a4cc98ff3770be6eb6f1c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:39:13.452032 27505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:13.447638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36766 (local address 127.25.71.67:35633)
0928 16:39:13.447783 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:13.447789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:13.447809 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:39:13.447977 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:13.447980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:13.448056 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:13.448181 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:13.448740 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.449353 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.450371 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.450606 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.450729 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:13.451123 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:13.451607 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:13.451705 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:13.451798 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 16:39:13.451861 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:39:13.466473 27556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:13.463033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:39791 (local address 127.25.71.67:49869)
0928 16:39:13.463372 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:13.463389 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:13.463504 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:13.463739 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:13.463742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:13.463749 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:39:13.463876 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:13.463881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.464595 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.464599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:13.465278 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:13.465288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.465714 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.465717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:13.465811 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:13.465868 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:39:13.466157 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:39:13.466164 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 16:39:13.466235 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 16:39:13.466315 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":100,"threads_started":1}
I20260928 16:39:13.466738 27503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:13.463163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:49869 (local address 127.25.71.65:39791)
0928 16:39:13.463442 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:13.463447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:13.463462 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:39:13.463608 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:13.463611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:13.463660 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:13.463719 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:13.463983 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.464495 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.465388 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.465608 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.465694 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:13.465999 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:13.466080 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:13.466165 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:13.466360 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 16:39:13.466490 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260928 16:39:13.475422 27297 raft_consensus.cc:1261] T fe9fff0a73f94306a3da314211151091 P 762e1920adbd485d87832a7faa7ed2d0 [term 1 FOLLOWER]: Refusing update from remote peer e465f4d9b62a4cc98ff3770be6eb6f1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:13.475683 27157 raft_consensus.cc:1261] T fe9fff0a73f94306a3da314211151091 P 822bbcfb122f43528736000f607ffdfd [term 1 FOLLOWER]: Refusing update from remote peer e465f4d9b62a4cc98ff3770be6eb6f1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:13.476460 27537 consensus_queue.cc:1035] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "762e1920adbd485d87832a7faa7ed2d0" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:13.476723 27514 consensus_queue.cc:1035] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "822bbcfb122f43528736000f607ffdfd" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 39791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:13.481161 27522 mvcc.cc:205] Tried to move back new op lower bound from 7334353115029848064 to 7334353114103189504. Current Snapshot: MvccSnapshot[applied={T|T < 7334353115029848064}]
I20260928 16:39:13.504655 27505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:13.499208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:44417 (local address 127.25.71.67:35633)
0928 16:39:13.499330 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:13.499334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:13.499654 (+ 320us) server_negotiation.cc:407] Connection header received
0928 16:39:13.500093 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:13.500097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:13.500177 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:13.500269 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:13.500584 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.501523 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.503010 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.503422 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.503603 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:13.504108 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:13.504209 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:13.504279 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:13.504402 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 16:39:13.504517 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 16:39:13.504738 27561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:13.499134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:35633 (local address 127.25.71.65:44417)
0928 16:39:13.499567 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:13.499584 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:13.499751 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:13.500324 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:13.500326 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:13.500335 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:39:13.500459 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:13.500468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.501649 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.501654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:13.502816 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:13.502828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.503723 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.503728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:13.503839 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:13.503918 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:39:13.504301 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:39:13.504307 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:39:13.504387 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 16:39:13.504510 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":162,"threads_started":1}
I20260928 16:39:13.540329 26999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:13.540542 26999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:13.543802 26999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f61f45b2b9f44faa0b09780f282feb9: Sending DeleteTablet for 3 replicas of tablet fe9fff0a73f94306a3da314211151091
I20260928 16:39:13.544734 27137 tablet_service.cc:1514] Processing DeleteTablet for tablet fe9fff0a73f94306a3da314211151091 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:13 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:54658
I20260928 16:39:13.544786 27277 tablet_service.cc:1514] Processing DeleteTablet for tablet fe9fff0a73f94306a3da314211151091 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:13 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:56952
I20260928 16:39:13.545018 27531 tablet_replica.cc:330] T fe9fff0a73f94306a3da314211151091 P 822bbcfb122f43528736000f607ffdfd: stopping tablet replica
I20260928 16:39:13.545199 27531 raft_consensus.cc:2229] T fe9fff0a73f94306a3da314211151091 P 822bbcfb122f43528736000f607ffdfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:13.545377 27531 raft_consensus.cc:2258] T fe9fff0a73f94306a3da314211151091 P 822bbcfb122f43528736000f607ffdfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:13.545533 27417 tablet_service.cc:1514] Processing DeleteTablet for tablet fe9fff0a73f94306a3da314211151091 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:13 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:36764
I20260928 16:39:13.545588 27530 tablet_replica.cc:330] T fe9fff0a73f94306a3da314211151091 P 762e1920adbd485d87832a7faa7ed2d0: stopping tablet replica
I20260928 16:39:13.545717 27530 raft_consensus.cc:2229] T fe9fff0a73f94306a3da314211151091 P 762e1920adbd485d87832a7faa7ed2d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:13.545761 27532 tablet_replica.cc:330] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c: stopping tablet replica
I20260928 16:39:13.545852 27530 raft_consensus.cc:2258] T fe9fff0a73f94306a3da314211151091 P 762e1920adbd485d87832a7faa7ed2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:13.545912 27532 raft_consensus.cc:2229] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:39:13.546270 27532 raft_consensus.cc:2258] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:13.547050 27530 ts_tablet_manager.cc:1905] T fe9fff0a73f94306a3da314211151091 P 762e1920adbd485d87832a7faa7ed2d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:13.547829 27531 ts_tablet_manager.cc:1905] T fe9fff0a73f94306a3da314211151091 P 822bbcfb122f43528736000f607ffdfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:13.550940 27531 ts_tablet_manager.cc:1918] T fe9fff0a73f94306a3da314211151091 P 822bbcfb122f43528736000f607ffdfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:39:13.550940 27530 ts_tablet_manager.cc:1918] T fe9fff0a73f94306a3da314211151091 P 762e1920adbd485d87832a7faa7ed2d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:39:13.551046 27531 log.cc:1196] T fe9fff0a73f94306a3da314211151091 P 822bbcfb122f43528736000f607ffdfd: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-0/wal/wals/fe9fff0a73f94306a3da314211151091
I20260928 16:39:13.551046 27530 log.cc:1196] T fe9fff0a73f94306a3da314211151091 P 762e1920adbd485d87832a7faa7ed2d0: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/fe9fff0a73f94306a3da314211151091
I20260928 16:39:13.551314 27531 ts_tablet_manager.cc:1939] T fe9fff0a73f94306a3da314211151091 P 822bbcfb122f43528736000f607ffdfd: Deleting consensus metadata
I20260928 16:39:13.551314 27530 ts_tablet_manager.cc:1939] T fe9fff0a73f94306a3da314211151091 P 762e1920adbd485d87832a7faa7ed2d0: Deleting consensus metadata
I20260928 16:39:13.551504 27532 ts_tablet_manager.cc:1905] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:13.552342 26986 catalog_manager.cc:4841] TS 762e1920adbd485d87832a7faa7ed2d0 (127.25.71.66:33913): tablet fe9fff0a73f94306a3da314211151091 (table test-table [id=09919e3e0a604b0cb0563a1911cd1495]) successfully deleted
I20260928 16:39:13.555224 26987 catalog_manager.cc:4841] TS 822bbcfb122f43528736000f607ffdfd (127.25.71.65:39791): tablet fe9fff0a73f94306a3da314211151091 (table test-table [id=09919e3e0a604b0cb0563a1911cd1495]) successfully deleted
I20260928 16:39:13.555820 27532 ts_tablet_manager.cc:1918] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:39:13.555979 27532 log.cc:1196] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/fe9fff0a73f94306a3da314211151091
I20260928 16:39:13.556381 27532 ts_tablet_manager.cc:1939] T fe9fff0a73f94306a3da314211151091 P e465f4d9b62a4cc98ff3770be6eb6f1c: Deleting consensus metadata
I20260928 16:39:13.557440 26985 catalog_manager.cc:4841] TS e465f4d9b62a4cc98ff3770be6eb6f1c (127.25.71.67:35633): tablet fe9fff0a73f94306a3da314211151091 (table test-table [id=09919e3e0a604b0cb0563a1911cd1495]) successfully deleted
Sep 28 16:39:13 dist-test-slave-nqgh krb5kdc[26955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613552, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.71.65@KRBTEST.COM
I20260928 16:39:13.567845 27503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:13.547537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54690 (local address 127.25.71.65:39791)
0928 16:39:13.547671 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:13.547676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:13.553082 (+ 5406us) server_negotiation.cc:407] Connection header received
0928 16:39:13.553351 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:13.553354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:13.553418 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:13.553473 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:13.554716 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.555527 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.556873 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.557258 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.557351 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:13.562286 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:13.562312 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:13.562344 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:13.562397 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:13.564963 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:13.565807 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:13.565816 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:13.565818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:13.565957 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:13.566295 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:13.566298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:13.566300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:13.566841 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:13.567139 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:13.567422 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 16:39:13.567620 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 16:39:13.569139 27197 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54690
I20260928 16:39:13.583215 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:13.571647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.25.71.126:44207)
0928 16:39:13.571867 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:13.571873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:13.572451 (+ 578us) server_negotiation.cc:407] Connection header received
0928 16:39:13.572647 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:13.572651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:13.572731 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:13.572845 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:13.573768 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.574710 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.576010 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:13.576461 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:13.576568 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:13.578736 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:13.578742 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:13.578745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:13.578795 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:13.580902 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:13.581588 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:13.581593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:13.581595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:13.581715 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:13.582025 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:13.582029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:13.582030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:13.582312 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:13.582454 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:13.582834 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 16:39:13.583004 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 16:39:13.584069 26999 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37786
I20260928 16:39:13.586530 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 27072
I20260928 16:39:13.597879 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 27212
I20260928 16:39:13.604439 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 27352
I20260928 16:39:13.611789 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 26968
2026-09-28T16:39:13Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3071 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:39:13 dist-test-slave-nqgh krb5kdc[27571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:39:13 dist-test-slave-nqgh krb5kdc[27571](info): set up 2 sockets
Sep 28 16:39:13 dist-test-slave-nqgh krb5kdc[27571](info): commencing operation
krb5kdc: starting...
W20260928 16:39:14.604278 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:39:14 dist-test-slave-nqgh krb5kdc[27571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:39:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:39:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:39:14.750922 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:40891
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:40891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:14.867185 27584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:14.867492 27584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:14.867552 27584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:14.871717 27584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:14.871809 27584 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 16:39:14.871833 27584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:14.871856 27584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:14.871877 27584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:14.871899 27584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:39:14.877537 27584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:40891
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:40891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:14.878805 27584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:14.879889 27584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:14.889626 27592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:14.889638 27589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:14.889683 27590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:14.890188 27584 server_base.cc:1034] running on GCE node
I20260928 16:39:14.891111 27584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:14.892726 27584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:14.893955 27584 hybrid_clock.cc:648] HybridClock initialized: now 1790613554893916 us; error 59 us; skew 500 ppm
Sep 28 16:39:14 dist-test-slave-nqgh krb5kdc[27571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613554, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:14.897720 27584 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:39:14.899418 27584 webserver.cc:460] Webserver started at http://127.25.71.126:42189/ using document root <none> and password file <none>
I20260928 16:39:14.900339 27584 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:14.900615 27584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:14.900936 27584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:14.906591 27584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "c5109fff1b9143c6817e5db99a9fbf37"
format_stamp: "Formatted at 2026-09-28 16:39:14 on dist-test-slave-nqgh"
server_key: "bfdcbf2c0dc1a32cab5a203ed83b8269"
server_key_iv: "8e4de30d49d21f542e7fcbcdfa056a6b"
server_key_version: "encryptionkey@0"
I20260928 16:39:14.908021 27584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "c5109fff1b9143c6817e5db99a9fbf37"
format_stamp: "Formatted at 2026-09-28 16:39:14 on dist-test-slave-nqgh"
server_key: "bfdcbf2c0dc1a32cab5a203ed83b8269"
server_key_iv: "8e4de30d49d21f542e7fcbcdfa056a6b"
server_key_version: "encryptionkey@0"
I20260928 16:39:14.915171 27584 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260928 16:39:14.919795 27599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:14.922084 27584 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:14.922320 27584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "c5109fff1b9143c6817e5db99a9fbf37"
format_stamp: "Formatted at 2026-09-28 16:39:14 on dist-test-slave-nqgh"
server_key: "bfdcbf2c0dc1a32cab5a203ed83b8269"
server_key_iv: "8e4de30d49d21f542e7fcbcdfa056a6b"
server_key_version: "encryptionkey@0"
I20260928 16:39:14.922516 27584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:14.948738 27584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:14.953161 27584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:14.953413 27584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:14.964100 27584 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:40891
I20260928 16:39:14.964217 27662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:40891 every 8 connection(s)
I20260928 16:39:14.965427 27584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:39:14.969044 27663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:14.970602 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 27584
I20260928 16:39:14.970793 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:39:14.971144 25885 external_mini_cluster.cc:1428] Setting key 95f6950627eb890681700a14f211a843
I20260928 16:39:14.977242 27663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37: Bootstrap starting.
I20260928 16:39:14.980352 27663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:14.981343 27663 log.cc:824] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:14.985075 27663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37: No bootstrap required, opened a new log
Sep 28 16:39:14 dist-test-slave-nqgh krb5kdc[27571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:14.991567 27663 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5109fff1b9143c6817e5db99a9fbf37" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 40891 } }
I20260928 16:39:14.992172 27663 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:14.992343 27663 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5109fff1b9143c6817e5db99a9fbf37, State: Initialized, Role: FOLLOWER
I20260928 16:39:14.993134 27663 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5109fff1b9143c6817e5db99a9fbf37" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 40891 } }
I20260928 16:39:14.993466 27663 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:39:14.993620 27663 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:39:14.993809 27663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:14.995561 27663 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5109fff1b9143c6817e5db99a9fbf37" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 40891 } }
I20260928 16:39:14.995879 27666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:14.973135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42326 (local address 127.25.71.126:40891)
0928 16:39:14.973805 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:14.973828 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:14.973885 (+ 57us) server_negotiation.cc:407] Connection header received
0928 16:39:14.974931 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:14.974959 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:14.975454 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:14.975873 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:14.976973 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:14.978237 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:14.980334 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:14.981154 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:14.981843 (+ 689us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:14.986619 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:14.986666 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:14.986705 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:14.986769 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:14.989808 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:14.990709 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:14.990720 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:14.990731 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:14.990964 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:14.991412 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:14.991415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:14.991418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:14.992295 (+ 877us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:14.993598 (+ 1303us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:14.993834 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 16:39:14.994340 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":178,"threads_started":1}
I20260928 16:39:14.996378 27663 leader_election.cc:302] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5109fff1b9143c6817e5db99a9fbf37; no voters:
I20260928 16:39:14.996951 27663 leader_election.cc:288] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:39:14.997043 27668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:14.997249 27668 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37 [term 1 LEADER]: Becoming Leader. State: Replica: c5109fff1b9143c6817e5db99a9fbf37, State: Running, Role: LEADER
I20260928 16:39:14.997617 27668 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5109fff1b9143c6817e5db99a9fbf37" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 40891 } }
I20260928 16:39:14.999356 27669 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5109fff1b9143c6817e5db99a9fbf37. Latest consensus state: current_term: 1 leader_uuid: "c5109fff1b9143c6817e5db99a9fbf37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5109fff1b9143c6817e5db99a9fbf37" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 40891 } } }
I20260928 16:39:14.999522 27669 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:14.999853 27670 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5109fff1b9143c6817e5db99a9fbf37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5109fff1b9143c6817e5db99a9fbf37" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 40891 } } }
I20260928 16:39:14.999950 27670 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:15.000350 27663 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:15.000535 27672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:39:15.004196 27672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:39:15.012976 27672 catalog_manager.cc:1347] Generated new cluster ID: cbba9d8c7914424588d7aeb2f6a86705
I20260928 16:39:15.013074 27672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:39:15.024041 27672 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:39:15.025163 27672 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:39:15.032269 27672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37: Generated new TSK 0
I20260928 16:39:15.033088 27672 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
I20260928 16:39:15.071395 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:40891
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:15.191862 27689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:15.192397 27689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:15.192538 27689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:15.197070 27689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:15.197166 27689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:15.197209 27689 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 16:39:15.197223 27689 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:39:15.197257 27689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:39:15.203887 27689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.71.126:40891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:15.205127 27689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:15.206673 27689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:15.218761 27697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:15.218847 27695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:15.218714 27694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:15.219352 27689 server_base.cc:1034] running on GCE node
I20260928 16:39:15.219983 27689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:15.220913 27689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:15.222146 27689 hybrid_clock.cc:648] HybridClock initialized: now 1790613555222114 us; error 48 us; skew 500 ppm
Sep 28 16:39:15 dist-test-slave-nqgh krb5kdc[27571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613555, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:15.226173 27689 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:39:15.227594 27689 webserver.cc:460] Webserver started at http://127.25.71.65:35991/ using document root <none> and password file <none>
I20260928 16:39:15.228405 27689 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:15.228495 27689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:15.228745 27689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:15.231760 27689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "2a30debbc5e345fa8095f643137028a0"
format_stamp: "Formatted at 2026-09-28 16:39:15 on dist-test-slave-nqgh"
server_key: "bc35d8ef3e2ae63f96bc2e6d6561af7d"
server_key_iv: "63c905a335476c4ec111deb6e7b5f64c"
server_key_version: "encryptionkey@0"
I20260928 16:39:15.232556 27689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "2a30debbc5e345fa8095f643137028a0"
format_stamp: "Formatted at 2026-09-28 16:39:15 on dist-test-slave-nqgh"
server_key: "bc35d8ef3e2ae63f96bc2e6d6561af7d"
server_key_iv: "63c905a335476c4ec111deb6e7b5f64c"
server_key_version: "encryptionkey@0"
I20260928 16:39:15.238051 27689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:39:15.242075 27705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:15.244200 27689 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:15.244414 27689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "2a30debbc5e345fa8095f643137028a0"
format_stamp: "Formatted at 2026-09-28 16:39:15 on dist-test-slave-nqgh"
server_key: "bc35d8ef3e2ae63f96bc2e6d6561af7d"
server_key_iv: "63c905a335476c4ec111deb6e7b5f64c"
server_key_version: "encryptionkey@0"
I20260928 16:39:15.244589 27689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:15.261552 27689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:15.266883 27689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:15.267138 27689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:15.269254 27689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:15.269332 27689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:15.269381 27689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:15.269397 27689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:39:15 dist-test-slave-nqgh krb5kdc[27571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613555, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:15.282641 27689 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:33657
I20260928 16:39:15.282688 27821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:33657 every 8 connection(s)
I20260928 16:39:15.283855 27689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:39:15.285768 27666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.270441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:46305 (local address 127.25.71.126:40891)
0928 16:39:15.270591 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:15.270595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:15.272420 (+ 1825us) server_negotiation.cc:407] Connection header received
0928 16:39:15.273928 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:15.273933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:15.274036 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:15.274146 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:15.275848 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.276470 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.277736 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.277993 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.278089 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.281264 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:15.281269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:15.281272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:15.281305 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:15.283137 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:15.283746 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:15.283749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:15.283751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:15.283831 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:15.284275 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:15.284278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:15.284280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:15.284551 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:15.284644 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:15.285400 (+ 756us) server_negotiation.cc:299] Negotiation successful
0928 16:39:15.285600 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:39:15.286550 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.271340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:40891 (local address 127.25.71.65:46305)
0928 16:39:15.272206 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:15.272283 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:15.273534 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:15.274340 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:15.274349 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:15.275009 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:15.275645 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.275662 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.276607 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.276611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.277610 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.277620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.278164 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.278168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.278356 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.279100 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:15.279124 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:15.281035 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:15.283277 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:15.283285 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:15.283308 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:15.283626 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:15.283979 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:15.283982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:15.283984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:15.284155 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:15.284700 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:15.284708 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:15.285024 (+ 316us) client_negotiation.cc:769] Sending connection context
0928 16:39:15.285207 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 16:39:15.285556 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":202,"threads_started":1}
I20260928 16:39:15.288384 27822 heartbeater.cc:347] Connected to a master server at 127.25.71.126:40891
I20260928 16:39:15.288584 27822 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:15.289176 27822 heartbeater.cc:511] Master 127.25.71.126:40891 requested a full tablet report, sending...
I20260928 16:39:15.291009 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 27689
I20260928 16:39:15.291141 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:39:15.291136 27614 ts_manager.cc:194] Registered new tserver with Master: 2a30debbc5e345fa8095f643137028a0 (127.25.71.65:33657)
I20260928 16:39:15.291473 25885 external_mini_cluster.cc:1428] Setting key 961ff2c51400cc15bc9604474f4b8557
I20260928 16:39:15.292670 27614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:46305
I20260928 16:39:15.301772 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.291938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:40891 (local address 127.25.71.65:36079)
0928 16:39:15.292397 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:15.292416 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:15.292537 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:15.292866 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:15.292869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:15.293219 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:15.293414 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.293427 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.294509 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.294515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.295443 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.295458 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.296183 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.296188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.296313 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.296990 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:15.297004 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:15.297530 (+ 526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:15.299608 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:15.299612 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:15.299615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:15.299889 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:15.300223 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:15.300227 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:15.300229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:15.300321 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:15.301034 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:15.301036 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:15.301164 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 16:39:15.301280 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 16:39:15.301522 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":185,"threads_started":1}
I20260928 16:39:15.301779 27666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.291826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:36079 (local address 127.25.71.126:40891)
0928 16:39:15.291979 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:15.291985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:15.292554 (+ 569us) server_negotiation.cc:407] Connection header received
0928 16:39:15.292642 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:15.292645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:15.292704 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:15.292810 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:15.293547 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.294315 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.295611 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.296004 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.296137 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.297732 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:15.297736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:15.297739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:15.297768 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:15.299456 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:15.299996 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:15.300000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:15.300002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:15.300135 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:15.300506 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:15.300513 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:15.300515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:15.300908 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:15.301057 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:15.301355 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 16:39:15.301577 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.25.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
I20260928 16:39:15.326967 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:40891
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:15.444114 27831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:15.444422 27831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:15.444474 27831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:15.448696 27831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:15.448776 27831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:15.448818 27831 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 16:39:15.448833 27831 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:39:15.448872 27831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:39:15.455287 27831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.71.126:40891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:15.456485 27831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:15.457554 27831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:15.469134 27837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:15.469066 27839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:15.469318 27831 server_base.cc:1034] running on GCE node
W20260928 16:39:15.469169 27836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:15.470072 27831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:15.471114 27831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:15.472345 27831 hybrid_clock.cc:648] HybridClock initialized: now 1790613555472307 us; error 58 us; skew 500 ppm
Sep 28 16:39:15 dist-test-slave-nqgh krb5kdc[27571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613555, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:15.476481 27831 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:39:15.478080 27831 webserver.cc:460] Webserver started at http://127.25.71.66:43753/ using document root <none> and password file <none>
I20260928 16:39:15.478960 27831 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:15.479080 27831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:15.479372 27831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:15.482252 27831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "ace0868b811e408ea26ba92547cf51ec"
format_stamp: "Formatted at 2026-09-28 16:39:15 on dist-test-slave-nqgh"
server_key: "ab08547596f91280e918f1d5c704a591"
server_key_iv: "06fe5aa55876634cf5422e97afce4daa"
server_key_version: "encryptionkey@0"
I20260928 16:39:15.483126 27831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "ace0868b811e408ea26ba92547cf51ec"
format_stamp: "Formatted at 2026-09-28 16:39:15 on dist-test-slave-nqgh"
server_key: "ab08547596f91280e918f1d5c704a591"
server_key_iv: "06fe5aa55876634cf5422e97afce4daa"
server_key_version: "encryptionkey@0"
I20260928 16:39:15.488610 27831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:39:15.492496 27846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:15.494465 27831 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:15.494697 27831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "ace0868b811e408ea26ba92547cf51ec"
format_stamp: "Formatted at 2026-09-28 16:39:15 on dist-test-slave-nqgh"
server_key: "ab08547596f91280e918f1d5c704a591"
server_key_iv: "06fe5aa55876634cf5422e97afce4daa"
server_key_version: "encryptionkey@0"
I20260928 16:39:15.494860 27831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:15.507812 27831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:15.511899 27831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:15.512163 27831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:15.514169 27831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:15.514237 27831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:15.514293 27831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:15.514325 27831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:39:15 dist-test-slave-nqgh krb5kdc[27571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613555, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:15.528268 27831 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:35315
I20260928 16:39:15.528331 27962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:35315 every 8 connection(s)
I20260928 16:39:15.529709 27831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:39:15.533384 27666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.515451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:49877 (local address 127.25.71.126:40891)
0928 16:39:15.515593 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:15.515597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:15.517212 (+ 1615us) server_negotiation.cc:407] Connection header received
0928 16:39:15.518517 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:15.518520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:15.518583 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:15.518788 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:15.520299 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.520936 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.523761 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.524044 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.524151 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.528191 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:15.528196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:15.528223 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:15.528268 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:15.530156 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:15.531123 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:15.531128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:15.531130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:15.531262 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:15.531786 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:15.531789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:15.531791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:15.532173 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:15.532295 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:15.533077 (+ 782us) server_negotiation.cc:299] Negotiation successful
0928 16:39:15.533227 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 16:39:15.534242 27855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.516225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:40891 (local address 127.25.71.66:49877)
0928 16:39:15.516991 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:15.517076 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:15.518283 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:15.519020 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:15.519028 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:15.519520 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:15.520120 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.520144 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.521145 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.521150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.523450 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.523483 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.524258 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.524264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.524525 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.525405 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:15.525434 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:15.527822 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:15.530376 (+ 2554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:15.530386 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:15.530407 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:15.530962 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:15.531361 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:15.531365 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:15.531367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:15.531579 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:15.532363 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:15.532370 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:15.532727 (+ 357us) client_negotiation.cc:769] Sending connection context
0928 16:39:15.532910 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 16:39:15.533273 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":193,"threads_started":1}
I20260928 16:39:15.535562 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 27831
I20260928 16:39:15.535714 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:39:15.536065 27963 heartbeater.cc:347] Connected to a master server at 127.25.71.126:40891
I20260928 16:39:15.536154 25885 external_mini_cluster.cc:1428] Setting key 81227e5fbcd338aac332dbffed2e8fbb
I20260928 16:39:15.536264 27963 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:15.536902 27963 heartbeater.cc:511] Master 127.25.71.126:40891 requested a full tablet report, sending...
I20260928 16:39:15.538190 27614 ts_manager.cc:194] Registered new tserver with Master: ace0868b811e408ea26ba92547cf51ec (127.25.71.66:35315)
I20260928 16:39:15.539080 27614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:49877
WARNING: no policy specified for kudu/127.25.71.67@KRBTEST.COM; defaulting to no policy
I20260928 16:39:15.550325 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.539667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:40891 (local address 127.25.71.66:38715)
0928 16:39:15.541268 (+ 1601us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:15.541286 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:15.541420 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:15.541757 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:15.541760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:15.542012 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:15.542153 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.542161 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.543353 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.543357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.544298 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.544314 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.544877 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.544881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.544991 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.545598 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:15.545614 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:15.546101 (+ 487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:15.548490 (+ 2389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:15.548493 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:15.548495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:15.548727 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:15.549100 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:15.549103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:15.549104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:15.549191 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:15.549762 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:15.549764 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:15.549872 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 16:39:15.550007 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 16:39:15.550149 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1476,"thread_start_us":120,"threads_started":1}
I20260928 16:39:15.550489 27666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.540064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:38715 (local address 127.25.71.126:40891)
0928 16:39:15.540199 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:15.540203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:15.541407 (+ 1204us) server_negotiation.cc:407] Connection header received
0928 16:39:15.541547 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:15.541550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:15.541608 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:15.541699 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:15.542302 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.542986 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.544461 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.544728 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.544829 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.546318 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:15.546321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:15.546324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:15.546356 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:15.548295 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:15.548877 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:15.548881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:15.548883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:15.548995 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:15.549286 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:15.549294 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:15.549295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:15.549658 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:15.549750 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:15.550165 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 16:39:15.550306 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
I20260928 16:39:15.573020 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:40891
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:15.695521 27972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:15.695828 27972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:15.695899 27972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:15.700126 27972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:15.700227 27972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:15.700289 27972 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 16:39:15.700313 27972 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:39:15.700371 27972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:39:15.706523 27972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.71.126:40891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:15.707868 27972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:15.709014 27972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:39:15.721326 27972 server_base.cc:1034] running on GCE node
W20260928 16:39:15.721357 27977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:15.721249 27980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:15.721225 27978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:15.722090 27972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:15.722934 27972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:15.724159 27972 hybrid_clock.cc:648] HybridClock initialized: now 1790613555724141 us; error 49 us; skew 500 ppm
Sep 28 16:39:15 dist-test-slave-nqgh krb5kdc[27571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613555, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:15.727684 27972 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:39:15.729158 27972 webserver.cc:460] Webserver started at http://127.25.71.67:40277/ using document root <none> and password file <none>
I20260928 16:39:15.729853 27972 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:15.729944 27972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:15.730161 27972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:15.732540 27972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "269ac13e0d0343a894d0bb87a2ea673c"
format_stamp: "Formatted at 2026-09-28 16:39:15 on dist-test-slave-nqgh"
server_key: "c6f7eba58c6018eb4f70f9b0577ac3e2"
server_key_iv: "6b9b65ee4166a899bfed4fb5334dc193"
server_key_version: "encryptionkey@0"
I20260928 16:39:15.733139 27972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "269ac13e0d0343a894d0bb87a2ea673c"
format_stamp: "Formatted at 2026-09-28 16:39:15 on dist-test-slave-nqgh"
server_key: "c6f7eba58c6018eb4f70f9b0577ac3e2"
server_key_iv: "6b9b65ee4166a899bfed4fb5334dc193"
server_key_version: "encryptionkey@0"
I20260928 16:39:15.737812 27972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 16:39:15.741117 27987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:15.743213 27972 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 16:39:15.743369 27972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "269ac13e0d0343a894d0bb87a2ea673c"
format_stamp: "Formatted at 2026-09-28 16:39:15 on dist-test-slave-nqgh"
server_key: "c6f7eba58c6018eb4f70f9b0577ac3e2"
server_key_iv: "6b9b65ee4166a899bfed4fb5334dc193"
server_key_version: "encryptionkey@0"
I20260928 16:39:15.743522 27972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:15.756340 27972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:15.760257 27972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:15.760452 27972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:15.762189 27972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:15.762264 27972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:15.762318 27972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:15.762338 27972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:15.773695 27972 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:41515
I20260928 16:39:15.773933 28103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:41515 every 8 connection(s)
I20260928 16:39:15.775360 27972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
Sep 28 16:39:15 dist-test-slave-nqgh krb5kdc[27571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613555, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:15.782142 27666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.763457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:52353 (local address 127.25.71.126:40891)
0928 16:39:15.763662 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:15.763668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:15.764957 (+ 1289us) server_negotiation.cc:407] Connection header received
0928 16:39:15.767308 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:15.767313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:15.767457 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:15.769681 (+ 2224us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:15.769836 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.770661 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.772694 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.773096 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.773214 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.777133 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:15.777138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:15.777142 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:15.777188 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:15.779076 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:15.779911 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:15.779915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:15.779916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:15.780024 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:15.780540 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:15.780543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:15.780544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:15.780805 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:15.780917 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:15.781782 (+ 865us) server_negotiation.cc:299] Negotiation successful
0928 16:39:15.781994 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 16:39:15.782881 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 27972
I20260928 16:39:15.783010 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:39:15.783329 25885 external_mini_cluster.cc:1428] Setting key ecddc18fa64a32c1655ad39a7d50e9c8
I20260928 16:39:15.783533 27997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.764040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:40891 (local address 127.25.71.67:52353)
0928 16:39:15.764759 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:15.764823 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:15.766718 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:15.767928 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:15.767942 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:15.768574 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:15.769467 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.769491 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.770842 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.770846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.772500 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.772518 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.773472 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.773476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.773731 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.774594 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:15.774620 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:15.776812 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:15.779290 (+ 2478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:15.779298 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:15.779318 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:15.779722 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:15.780168 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:15.780171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:15.780173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:15.780396 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:15.781000 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:15.781011 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:15.781449 (+ 438us) client_negotiation.cc:769] Sending connection context
0928 16:39:15.781698 (+ 249us) client_negotiation.cc:240] Negotiation successful
0928 16:39:15.782113 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":157,"threads_started":1}
I20260928 16:39:15.785411 28104 heartbeater.cc:347] Connected to a master server at 127.25.71.126:40891
I20260928 16:39:15.785633 28104 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:15.786350 28104 heartbeater.cc:511] Master 127.25.71.126:40891 requested a full tablet report, sending...
I20260928 16:39:15.787640 27614 ts_manager.cc:194] Registered new tserver with Master: 269ac13e0d0343a894d0bb87a2ea673c (127.25.71.67:41515)
I20260928 16:39:15.788313 27614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:52353
I20260928 16:39:15.797864 28110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.788639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:40891 (local address 127.25.71.67:39283)
0928 16:39:15.789060 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:15.789077 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:15.789238 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:15.789514 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:15.789517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:15.789825 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:15.789966 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.789975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.790922 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.790926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.791693 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.791704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.792390 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.792393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.792492 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.793238 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:15.793252 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:15.793669 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:15.795729 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:15.795733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:15.795735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:15.796019 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:15.796478 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:15.796481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:15.796483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:15.796617 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:15.797176 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:15.797180 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:15.797307 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 16:39:15.797433 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 16:39:15.797647 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":154,"threads_started":1}
I20260928 16:39:15.797951 27666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.788728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:39283 (local address 127.25.71.126:40891)
0928 16:39:15.788984 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:15.788992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:15.789199 (+ 207us) server_negotiation.cc:407] Connection header received
0928 16:39:15.789352 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:15.789354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:15.789401 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:15.789464 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:15.790121 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.790763 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.791981 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.792218 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.792319 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.793849 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:15.793852 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:15.793854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:15.793955 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:15.795566 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:15.796237 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:15.796240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:15.796242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:15.796336 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:15.796774 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:15.796777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:15.796779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:15.797014 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:15.797135 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:15.797586 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 16:39:15.797786 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 16:39:15.799255 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:39:15 dist-test-slave-nqgh krb5kdc[27571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:39:15 dist-test-slave-nqgh krb5kdc[27571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:15.831532 27666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.820892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42328 (local address 127.25.71.126:40891)
0928 16:39:15.821086 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:15.821090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:15.821105 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:39:15.821149 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:15.821151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:15.821193 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:15.821271 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:15.822119 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.822688 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.823571 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.823787 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.823885 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.826978 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:15.826983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:15.826985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:15.827021 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:15.828829 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:15.829629 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:15.829633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:15.829634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:15.829746 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:15.830221 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:15.830229 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:15.830230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:15.830612 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:15.830787 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:15.831149 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 16:39:15.831295 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 16:39:15.836159 27616 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:39:15.839519 27616 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:39:15.857553 28122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.850881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45126 (local address 127.25.71.67:41515)
0928 16:39:15.851804 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:15.851818 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:15.851848 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:39:15.851956 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:15.851967 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:15.852249 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:15.852434 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:15.852776 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.854055 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.855848 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.856677 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.856789 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.857009 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:15.857223 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 16:39:15.857351 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":179,"threads_started":1}
I20260928 16:39:15.857553 28121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.850697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:41515 (local address 127.0.0.1:45126)
0928 16:39:15.851130 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:15.851181 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:15.851312 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:15.852432 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:15.852439 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:15.852467 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:15.852643 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.852651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.854194 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.854197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.855701 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.855713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.856813 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.856817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.856989 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.857046 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 16:39:15.857176 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 16:39:15.857354 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":2,"thread_start_us":130,"threads_started":1}
I20260928 16:39:15.857807 28123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.851185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:33657 (local address 127.0.0.1:47390)
0928 16:39:15.851694 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:15.851712 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:15.851855 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:15.853143 (+ 1288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:15.853146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:15.853163 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:15.853274 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.853281 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.854610 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.854614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.856172 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.856186 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.857308 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.857312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.857409 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.857431 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 16:39:15.857470 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 16:39:15.857531 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":354,"thread_start_us":115,"threads_started":1}
I20260928 16:39:15.858486 28126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.852377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47390 (local address 127.25.71.65:33657)
0928 16:39:15.852698 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:15.852703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:15.852724 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:39:15.852803 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:15.852808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:15.852989 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:15.853143 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:15.853393 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.854442 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.856364 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.857172 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.858010 (+ 838us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.858183 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:15.858239 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 16:39:15.858346 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":98,"threads_started":1}
I20260928 16:39:15.858779 28124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.851513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:35315 (local address 127.0.0.1:45868)
0928 16:39:15.852740 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:15.852753 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:15.852846 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:15.853408 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:15.853411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:15.853427 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:15.853525 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.853530 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.854690 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.854694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.855939 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.855947 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.857215 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.857219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.857329 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.857361 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 16:39:15.857428 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 16:39:15.857531 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1161,"mutex_wait_us":48,"thread_start_us":113,"threads_started":1}
I20260928 16:39:15.861634 28037 tablet_service.cc:1467] Processing CreateTablet for tablet f6883603e4674f218eb3eedeae94e16b (DEFAULT_TABLE table=test-table [id=eb9ad4cc7b84448ea80e594f7cc03a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:15.862687 28125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.851988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45868 (local address 127.25.71.66:35315)
0928 16:39:15.852293 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:15.852304 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:15.852822 (+ 518us) server_negotiation.cc:407] Connection header received
0928 16:39:15.852947 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:15.852954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:15.853145 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:15.853350 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:15.853626 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.854516 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.856054 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.856839 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.856950 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.857141 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:15.857625 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 16:39:15.862437 (+ 4812us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":102,"threads_started":1}
I20260928 16:39:15.862900 28037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6883603e4674f218eb3eedeae94e16b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:15.862303 27755 tablet_service.cc:1467] Processing CreateTablet for tablet f6883603e4674f218eb3eedeae94e16b (DEFAULT_TABLE table=test-table [id=eb9ad4cc7b84448ea80e594f7cc03a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:15.864146 27755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6883603e4674f218eb3eedeae94e16b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:15.867231 27896 tablet_service.cc:1467] Processing CreateTablet for tablet f6883603e4674f218eb3eedeae94e16b (DEFAULT_TABLE table=test-table [id=eb9ad4cc7b84448ea80e594f7cc03a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:15.869159 27896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6883603e4674f218eb3eedeae94e16b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:15.871845 28127 tablet_bootstrap.cc:492] T f6883603e4674f218eb3eedeae94e16b P 269ac13e0d0343a894d0bb87a2ea673c: Bootstrap starting.
I20260928 16:39:15.874300 28128 tablet_bootstrap.cc:492] T f6883603e4674f218eb3eedeae94e16b P 2a30debbc5e345fa8095f643137028a0: Bootstrap starting.
I20260928 16:39:15.875649 28127 tablet_bootstrap.cc:654] T f6883603e4674f218eb3eedeae94e16b P 269ac13e0d0343a894d0bb87a2ea673c: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:15.876603 28127 log.cc:824] T f6883603e4674f218eb3eedeae94e16b P 269ac13e0d0343a894d0bb87a2ea673c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:15.876767 28128 tablet_bootstrap.cc:654] T f6883603e4674f218eb3eedeae94e16b P 2a30debbc5e345fa8095f643137028a0: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:15.877461 28128 log.cc:824] T f6883603e4674f218eb3eedeae94e16b P 2a30debbc5e345fa8095f643137028a0: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:15.879205 28127 tablet_bootstrap.cc:492] T f6883603e4674f218eb3eedeae94e16b P 269ac13e0d0343a894d0bb87a2ea673c: No bootstrap required, opened a new log
I20260928 16:39:15.879516 28128 tablet_bootstrap.cc:492] T f6883603e4674f218eb3eedeae94e16b P 2a30debbc5e345fa8095f643137028a0: No bootstrap required, opened a new log
I20260928 16:39:15.879511 28127 ts_tablet_manager.cc:1397] T f6883603e4674f218eb3eedeae94e16b P 269ac13e0d0343a894d0bb87a2ea673c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 16:39:15.879472 28131 tablet_bootstrap.cc:492] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec: Bootstrap starting.
I20260928 16:39:15.879688 28128 ts_tablet_manager.cc:1397] T f6883603e4674f218eb3eedeae94e16b P 2a30debbc5e345fa8095f643137028a0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:39:15.882447 28131 tablet_bootstrap.cc:654] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:15.883636 28131 log.cc:824] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:15.885023 28128 raft_consensus.cc:348] T f6883603e4674f218eb3eedeae94e16b P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } }
I20260928 16:39:15.885025 28127 raft_consensus.cc:348] T f6883603e4674f218eb3eedeae94e16b P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } }
I20260928 16:39:15.885386 28127 raft_consensus.cc:374] T f6883603e4674f218eb3eedeae94e16b P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:15.885386 28128 raft_consensus.cc:374] T f6883603e4674f218eb3eedeae94e16b P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:15.885457 28127 raft_consensus.cc:729] T f6883603e4674f218eb3eedeae94e16b P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269ac13e0d0343a894d0bb87a2ea673c, State: Initialized, Role: FOLLOWER
I20260928 16:39:15.885465 28128 raft_consensus.cc:729] T f6883603e4674f218eb3eedeae94e16b P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a30debbc5e345fa8095f643137028a0, State: Initialized, Role: FOLLOWER
I20260928 16:39:15.886049 28128 consensus_queue.cc:260] T f6883603e4674f218eb3eedeae94e16b P 2a30debbc5e345fa8095f643137028a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } }
I20260928 16:39:15.886049 28127 consensus_queue.cc:260] T f6883603e4674f218eb3eedeae94e16b P 269ac13e0d0343a894d0bb87a2ea673c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } }
I20260928 16:39:15.886612 28131 tablet_bootstrap.cc:492] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec: No bootstrap required, opened a new log
I20260928 16:39:15.887092 28104 heartbeater.cc:503] Master 127.25.71.126:40891 was elected leader, sending a full tablet report...
I20260928 16:39:15.887296 28128 ts_tablet_manager.cc:1428] T f6883603e4674f218eb3eedeae94e16b P 2a30debbc5e345fa8095f643137028a0: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260928 16:39:15.887296 28127 ts_tablet_manager.cc:1428] T f6883603e4674f218eb3eedeae94e16b P 269ac13e0d0343a894d0bb87a2ea673c: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260928 16:39:15.887765 27822 heartbeater.cc:503] Master 127.25.71.126:40891 was elected leader, sending a full tablet report...
I20260928 16:39:15.887853 28131 ts_tablet_manager.cc:1397] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 16:39:15.891948 28131 raft_consensus.cc:348] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } }
I20260928 16:39:15.892304 28131 raft_consensus.cc:374] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:15.892383 28131 raft_consensus.cc:729] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ace0868b811e408ea26ba92547cf51ec, State: Initialized, Role: FOLLOWER
I20260928 16:39:15.892874 28131 consensus_queue.cc:260] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } }
I20260928 16:39:15.894066 28131 ts_tablet_manager.cc:1428] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:39:15.894093 27963 heartbeater.cc:503] Master 127.25.71.126:40891 was elected leader, sending a full tablet report...
I20260928 16:39:15.920478 28135 raft_consensus.cc:482] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:39:15.920733 28135 raft_consensus.cc:504] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } }
I20260928 16:39:15.922045 28135 leader_election.cc:288] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269ac13e0d0343a894d0bb87a2ea673c (127.25.71.67:41515), 2a30debbc5e345fa8095f643137028a0 (127.25.71.65:33657)
I20260928 16:39:15.927016 28126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.922647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:60065 (local address 127.25.71.65:33657)
0928 16:39:15.922801 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:15.922805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:15.922820 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:39:15.923003 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:15.923007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:15.923077 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:15.923190 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:15.923479 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.924128 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.925673 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.926288 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.926436 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.926508 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:15.926740 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 16:39:15.926837 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:39:15.927102 27855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.922434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:33657 (local address 127.25.71.66:60065)
0928 16:39:15.922679 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:15.922695 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:15.922830 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:15.923218 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:15.923221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:15.923237 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:15.923354 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.923359 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.924280 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.924283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.925504 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.925517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.926452 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.926455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.926567 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.926595 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 16:39:15.926722 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 16:39:15.926914 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260928 16:39:15.927217 28122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.922663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:38371 (local address 127.25.71.67:41515)
0928 16:39:15.922860 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:15.922866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:15.923198 (+ 332us) server_negotiation.cc:407] Connection header received
0928 16:39:15.923268 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:15.923270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:15.923321 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:15.923384 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:15.924036 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.924622 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.925961 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.926564 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.926718 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.926769 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:15.926919 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 16:39:15.927000 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 16:39:15.927100 28136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.922466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:41515 (local address 127.25.71.66:38371)
0928 16:39:15.923037 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:15.923049 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:15.923137 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:15.923423 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:15.923426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:15.923442 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:15.923903 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.923910 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.924762 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.924766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.925826 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.925834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.926701 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.926705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.926795 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.926816 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 16:39:15.926895 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 16:39:15.926953 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":226,"threads_started":1}
I20260928 16:39:15.928171 28046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6883603e4674f218eb3eedeae94e16b" candidate_uuid: "ace0868b811e408ea26ba92547cf51ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269ac13e0d0343a894d0bb87a2ea673c" is_pre_election: true
I20260928 16:39:15.928205 27775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6883603e4674f218eb3eedeae94e16b" candidate_uuid: "ace0868b811e408ea26ba92547cf51ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a30debbc5e345fa8095f643137028a0" is_pre_election: true
I20260928 16:39:15.928597 28046 raft_consensus.cc:2454] T f6883603e4674f218eb3eedeae94e16b P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ace0868b811e408ea26ba92547cf51ec in term 0.
I20260928 16:39:15.928632 27775 raft_consensus.cc:2454] T f6883603e4674f218eb3eedeae94e16b P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ace0868b811e408ea26ba92547cf51ec in term 0.
I20260928 16:39:15.929210 27850 leader_election.cc:302] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 269ac13e0d0343a894d0bb87a2ea673c, 2a30debbc5e345fa8095f643137028a0, ace0868b811e408ea26ba92547cf51ec; no voters:
I20260928 16:39:15.929903 28135 raft_consensus.cc:2781] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:39:15.929986 28135 raft_consensus.cc:482] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:39:15.930029 28135 raft_consensus.cc:3037] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:15.931573 28135 raft_consensus.cc:504] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } }
I20260928 16:39:15.932060 28135 leader_election.cc:288] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec [CANDIDATE]: Term 1 election: Requested vote from peers 269ac13e0d0343a894d0bb87a2ea673c (127.25.71.67:41515), 2a30debbc5e345fa8095f643137028a0 (127.25.71.65:33657)
I20260928 16:39:15.932718 28046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6883603e4674f218eb3eedeae94e16b" candidate_uuid: "ace0868b811e408ea26ba92547cf51ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269ac13e0d0343a894d0bb87a2ea673c"
I20260928 16:39:15.932734 27775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6883603e4674f218eb3eedeae94e16b" candidate_uuid: "ace0868b811e408ea26ba92547cf51ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a30debbc5e345fa8095f643137028a0"
I20260928 16:39:15.932920 28046 raft_consensus.cc:3037] T f6883603e4674f218eb3eedeae94e16b P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:15.932921 27775 raft_consensus.cc:3037] T f6883603e4674f218eb3eedeae94e16b P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:15.934410 28046 raft_consensus.cc:2454] T f6883603e4674f218eb3eedeae94e16b P 269ac13e0d0343a894d0bb87a2ea673c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ace0868b811e408ea26ba92547cf51ec in term 1.
I20260928 16:39:15.934410 27775 raft_consensus.cc:2454] T f6883603e4674f218eb3eedeae94e16b P 2a30debbc5e345fa8095f643137028a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ace0868b811e408ea26ba92547cf51ec in term 1.
I20260928 16:39:15.934923 27850 leader_election.cc:302] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a30debbc5e345fa8095f643137028a0, ace0868b811e408ea26ba92547cf51ec; no voters:
I20260928 16:39:15.935285 28135 raft_consensus.cc:2781] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:15.935612 28135 raft_consensus.cc:686] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec [term 1 LEADER]: Becoming Leader. State: Replica: ace0868b811e408ea26ba92547cf51ec, State: Running, Role: LEADER
I20260928 16:39:15.936076 28135 consensus_queue.cc:237] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } }
I20260928 16:39:15.940937 27614 catalog_manager.cc:5495] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec reported cstate change: term changed from 0 to 1, leader changed from <none> to ace0868b811e408ea26ba92547cf51ec (127.25.71.66). New cstate: current_term: 1 leader_uuid: "ace0868b811e408ea26ba92547cf51ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } health_report { overall_health: UNKNOWN } } }
Sep 28 16:39:15 dist-test-slave-nqgh krb5kdc[27571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613554, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:15.992775 27666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.981470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42330 (local address 127.25.71.126:40891)
0928 16:39:15.981698 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:15.981703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:15.981980 (+ 277us) server_negotiation.cc:407] Connection header received
0928 16:39:15.982054 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:15.982056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:15.982111 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:15.982207 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:15.983467 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.984178 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.985397 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.985638 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.985748 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.988892 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:15.988896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:15.988898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:15.988937 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:15.990534 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:15.991352 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:15.991355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:15.991356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:15.991478 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:15.991801 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:15.991805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:15.991807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:15.992036 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:15.992136 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:15.992447 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 16:39:15.992640 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":117}
I20260928 16:39:15.992805 28144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:15.981262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:40891 (local address 127.0.0.1:42330)
0928 16:39:15.981698 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:15.981715 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:15.981880 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:15.982316 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:15.982319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:15.983060 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:15.983253 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.983263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.984361 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.984364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.985219 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:15.985232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:15.985785 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:15.985788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:15.985878 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:15.986721 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:15.986735 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:15.988658 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:15.990731 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:15.990748 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:15.990757 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:15.991174 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:15.991600 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:15.991605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:15.991607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:15.991700 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:15.992151 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:15.992161 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:15.992277 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 16:39:15.992371 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 16:39:15.992567 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":156,"threads_started":1}
I20260928 16:39:15.998131 27614 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:42330:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 16:39:15.999094 27614 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:39:16.006125 27755 tablet_service.cc:1467] Processing CreateTablet for tablet d455497f3e9a4b4188be5af8627cd54f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=34ea65c217fc4bd0a63e8e2c220fdc3b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:39:16.007087 27755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d455497f3e9a4b4188be5af8627cd54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:16.006804 28037 tablet_service.cc:1467] Processing CreateTablet for tablet d455497f3e9a4b4188be5af8627cd54f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=34ea65c217fc4bd0a63e8e2c220fdc3b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:39:16.006409 27896 tablet_service.cc:1467] Processing CreateTablet for tablet d455497f3e9a4b4188be5af8627cd54f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=34ea65c217fc4bd0a63e8e2c220fdc3b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:39:16.007470 28037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d455497f3e9a4b4188be5af8627cd54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:16.007471 27896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d455497f3e9a4b4188be5af8627cd54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:16.010186 28128 tablet_bootstrap.cc:492] T d455497f3e9a4b4188be5af8627cd54f P 2a30debbc5e345fa8095f643137028a0: Bootstrap starting.
I20260928 16:39:16.011268 28128 tablet_bootstrap.cc:654] T d455497f3e9a4b4188be5af8627cd54f P 2a30debbc5e345fa8095f643137028a0: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:16.011597 28131 tablet_bootstrap.cc:492] T d455497f3e9a4b4188be5af8627cd54f P ace0868b811e408ea26ba92547cf51ec: Bootstrap starting.
I20260928 16:39:16.012689 28131 tablet_bootstrap.cc:654] T d455497f3e9a4b4188be5af8627cd54f P ace0868b811e408ea26ba92547cf51ec: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:16.012907 28128 tablet_bootstrap.cc:492] T d455497f3e9a4b4188be5af8627cd54f P 2a30debbc5e345fa8095f643137028a0: No bootstrap required, opened a new log
I20260928 16:39:16.013011 28128 ts_tablet_manager.cc:1397] T d455497f3e9a4b4188be5af8627cd54f P 2a30debbc5e345fa8095f643137028a0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:39:16.013461 28127 tablet_bootstrap.cc:492] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c: Bootstrap starting.
I20260928 16:39:16.013551 28128 raft_consensus.cc:348] T d455497f3e9a4b4188be5af8627cd54f P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:16.013677 28128 raft_consensus.cc:374] T d455497f3e9a4b4188be5af8627cd54f P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:16.013720 28128 raft_consensus.cc:729] T d455497f3e9a4b4188be5af8627cd54f P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a30debbc5e345fa8095f643137028a0, State: Initialized, Role: FOLLOWER
I20260928 16:39:16.013890 28128 consensus_queue.cc:260] T d455497f3e9a4b4188be5af8627cd54f P 2a30debbc5e345fa8095f643137028a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:16.014185 28128 ts_tablet_manager.cc:1428] T d455497f3e9a4b4188be5af8627cd54f P 2a30debbc5e345fa8095f643137028a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:39:16.014410 28131 tablet_bootstrap.cc:492] T d455497f3e9a4b4188be5af8627cd54f P ace0868b811e408ea26ba92547cf51ec: No bootstrap required, opened a new log
I20260928 16:39:16.014271 28133 tablet_replica.cc:439] T d455497f3e9a4b4188be5af8627cd54f P 2a30debbc5e345fa8095f643137028a0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } } }
I20260928 16:39:16.014516 28131 ts_tablet_manager.cc:1397] T d455497f3e9a4b4188be5af8627cd54f P ace0868b811e408ea26ba92547cf51ec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:39:16.014498 28133 tablet_replica.cc:442] T d455497f3e9a4b4188be5af8627cd54f P 2a30debbc5e345fa8095f643137028a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:39:16.014684 28127 tablet_bootstrap.cc:654] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:16.015137 28131 raft_consensus.cc:348] T d455497f3e9a4b4188be5af8627cd54f P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:16.015259 28131 raft_consensus.cc:374] T d455497f3e9a4b4188be5af8627cd54f P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:16.015296 28131 raft_consensus.cc:729] T d455497f3e9a4b4188be5af8627cd54f P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ace0868b811e408ea26ba92547cf51ec, State: Initialized, Role: FOLLOWER
I20260928 16:39:16.015446 28131 consensus_queue.cc:260] T d455497f3e9a4b4188be5af8627cd54f P ace0868b811e408ea26ba92547cf51ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:16.015815 28135 tablet_replica.cc:439] T d455497f3e9a4b4188be5af8627cd54f P ace0868b811e408ea26ba92547cf51ec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } } }
I20260928 16:39:16.015972 28135 tablet_replica.cc:442] T d455497f3e9a4b4188be5af8627cd54f P ace0868b811e408ea26ba92547cf51ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:39:16.016211 28127 tablet_bootstrap.cc:492] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c: No bootstrap required, opened a new log
I20260928 16:39:16.016321 28127 ts_tablet_manager.cc:1397] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:39:16.016892 28127 raft_consensus.cc:348] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:16.017023 28127 raft_consensus.cc:374] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:16.017057 28127 raft_consensus.cc:729] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269ac13e0d0343a894d0bb87a2ea673c, State: Initialized, Role: FOLLOWER
I20260928 16:39:16.017074 28131 ts_tablet_manager.cc:1428] T d455497f3e9a4b4188be5af8627cd54f P ace0868b811e408ea26ba92547cf51ec: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 16:39:16.017460 28127 consensus_queue.cc:260] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:16.017774 28127 ts_tablet_manager.cc:1428] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:39:16.017910 28134 tablet_replica.cc:439] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } } }
I20260928 16:39:16.018059 28134 tablet_replica.cc:442] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 16:39:16.028151 28105 tablet.cc:2369] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:39:16.031477 27964 tablet.cc:2369] T d455497f3e9a4b4188be5af8627cd54f P ace0868b811e408ea26ba92547cf51ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:39:16.035692 27823 tablet.cc:2369] T d455497f3e9a4b4188be5af8627cd54f P 2a30debbc5e345fa8095f643137028a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:39:16.113930 28134 raft_consensus.cc:482] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:39:16.114269 28134 raft_consensus.cc:504] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:16.115797 28134 leader_election.cc:288] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a30debbc5e345fa8095f643137028a0 (127.25.71.65:33657), ace0868b811e408ea26ba92547cf51ec (127.25.71.66:35315)
I20260928 16:39:16.121883 27997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:16.116155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:35315 (local address 127.25.71.67:51103)
0928 16:39:16.116371 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:16.116389 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:16.116507 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:16.117088 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:16.117092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:16.117117 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:16.117269 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:16.117277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.118532 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.118536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:16.120128 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:16.120145 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.121222 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.121227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:16.121400 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:16.121437 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 16:39:16.121540 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 16:39:16.121657 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":122}
I20260928 16:39:16.121918 28126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:16.116425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:56311 (local address 127.25.71.65:33657)
0928 16:39:16.116574 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:16.116579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:16.116871 (+ 292us) server_negotiation.cc:407] Connection header received
0928 16:39:16.116986 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:16.116990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:16.117049 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:16.117135 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:16.117548 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.118445 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.120321 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.121014 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.121224 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:16.121273 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:16.121633 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 16:39:16.121739 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:39:16.121918 28125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:16.116249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:51103 (local address 127.25.71.66:35315)
0928 16:39:16.116459 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:16.116464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:16.116482 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:39:16.116750 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:16.116754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:16.116867 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:16.117073 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:16.117438 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.118367 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.120321 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.121014 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.121196 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:16.121271 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:16.121626 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 16:39:16.121745 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 16:39:16.121894 28148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:16.116194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:33657 (local address 127.25.71.67:56311)
0928 16:39:16.116764 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:16.116777 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:16.116876 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:16.117261 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:16.117264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:16.117278 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:16.117409 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:16.117416 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.118574 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.118581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:16.120128 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:16.120145 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.121245 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.121249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:16.121384 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:16.121481 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 16:39:16.121552 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 16:39:16.121657 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":142,"thread_start_us":184,"threads_started":1}
I20260928 16:39:16.122840 27916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d455497f3e9a4b4188be5af8627cd54f" candidate_uuid: "269ac13e0d0343a894d0bb87a2ea673c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ace0868b811e408ea26ba92547cf51ec" is_pre_election: true
I20260928 16:39:16.122910 27775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d455497f3e9a4b4188be5af8627cd54f" candidate_uuid: "269ac13e0d0343a894d0bb87a2ea673c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a30debbc5e345fa8095f643137028a0" is_pre_election: true
I20260928 16:39:16.123145 27916 raft_consensus.cc:2454] T d455497f3e9a4b4188be5af8627cd54f P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269ac13e0d0343a894d0bb87a2ea673c in term 0.
I20260928 16:39:16.123201 27775 raft_consensus.cc:2454] T d455497f3e9a4b4188be5af8627cd54f P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269ac13e0d0343a894d0bb87a2ea673c in term 0.
I20260928 16:39:16.123714 27991 leader_election.cc:302] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269ac13e0d0343a894d0bb87a2ea673c, 2a30debbc5e345fa8095f643137028a0; no voters:
I20260928 16:39:16.124151 28134 raft_consensus.cc:2781] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:39:16.124235 28134 raft_consensus.cc:482] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:39:16.124276 28134 raft_consensus.cc:3037] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:16.125487 28134 raft_consensus.cc:504] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:16.125931 28134 leader_election.cc:288] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c [CANDIDATE]: Term 1 election: Requested vote from peers 2a30debbc5e345fa8095f643137028a0 (127.25.71.65:33657), ace0868b811e408ea26ba92547cf51ec (127.25.71.66:35315)
I20260928 16:39:16.126537 27916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d455497f3e9a4b4188be5af8627cd54f" candidate_uuid: "269ac13e0d0343a894d0bb87a2ea673c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ace0868b811e408ea26ba92547cf51ec"
I20260928 16:39:16.126533 27775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d455497f3e9a4b4188be5af8627cd54f" candidate_uuid: "269ac13e0d0343a894d0bb87a2ea673c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a30debbc5e345fa8095f643137028a0"
I20260928 16:39:16.126755 27775 raft_consensus.cc:3037] T d455497f3e9a4b4188be5af8627cd54f P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:16.126755 27916 raft_consensus.cc:3037] T d455497f3e9a4b4188be5af8627cd54f P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:16.128233 27775 raft_consensus.cc:2454] T d455497f3e9a4b4188be5af8627cd54f P 2a30debbc5e345fa8095f643137028a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269ac13e0d0343a894d0bb87a2ea673c in term 1.
I20260928 16:39:16.128233 27916 raft_consensus.cc:2454] T d455497f3e9a4b4188be5af8627cd54f P ace0868b811e408ea26ba92547cf51ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269ac13e0d0343a894d0bb87a2ea673c in term 1.
I20260928 16:39:16.128677 27991 leader_election.cc:302] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269ac13e0d0343a894d0bb87a2ea673c, 2a30debbc5e345fa8095f643137028a0; no voters:
I20260928 16:39:16.129046 28134 raft_consensus.cc:2781] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:16.129318 28134 raft_consensus.cc:686] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c [term 1 LEADER]: Becoming Leader. State: Replica: 269ac13e0d0343a894d0bb87a2ea673c, State: Running, Role: LEADER
I20260928 16:39:16.129772 28134 consensus_queue.cc:237] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:16.132320 28149 tablet_replica.cc:439] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c: TxnStatusTablet state changed. Reason: New leader 269ac13e0d0343a894d0bb87a2ea673c. Latest consensus state: current_term: 1 leader_uuid: "269ac13e0d0343a894d0bb87a2ea673c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } } }
I20260928 16:39:16.132519 28149 tablet_replica.cc:442] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:39:16.132948 28151 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 16:39:16.133200 28151 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 16:39:16.134163 27616 catalog_manager.cc:5495] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c reported cstate change: term changed from 0 to 1, leader changed from <none> to 269ac13e0d0343a894d0bb87a2ea673c (127.25.71.67). New cstate: current_term: 1 leader_uuid: "269ac13e0d0343a894d0bb87a2ea673c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } health_report { overall_health: HEALTHY } } }
I20260928 16:39:16.150091 28144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:16.145396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:41515 (local address 127.0.0.1:45132)
0928 16:39:16.145907 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:16.145929 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:16.146118 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:16.146518 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:16.146521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:16.146530 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:39:16.146725 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:16.146732 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.147610 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.147616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:16.148309 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:16.148323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.148834 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.148838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:16.148936 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:16.148998 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:39:16.149738 (+ 740us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:39:16.149755 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 16:39:16.149844 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 16:39:16.149952 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330}
I20260928 16:39:16.150125 28122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:16.145618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.25.71.67:41515)
0928 16:39:16.145868 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:16.145872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:16.146010 (+ 138us) server_negotiation.cc:407] Connection header received
0928 16:39:16.146289 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:16.146292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:16.146358 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:16.146479 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:16.146848 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.147430 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.148435 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.148701 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.148793 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:16.149148 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:16.149633 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:16.149738 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:16.149869 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 16:39:16.149938 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260928 16:39:16.156008 27775 raft_consensus.cc:1261] T d455497f3e9a4b4188be5af8627cd54f P 2a30debbc5e345fa8095f643137028a0 [term 1 FOLLOWER]: Refusing update from remote peer 269ac13e0d0343a894d0bb87a2ea673c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:16.156005 27916 raft_consensus.cc:1261] T d455497f3e9a4b4188be5af8627cd54f P ace0868b811e408ea26ba92547cf51ec [term 1 FOLLOWER]: Refusing update from remote peer 269ac13e0d0343a894d0bb87a2ea673c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:16.156754 28134 consensus_queue.cc:1035] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:16.157361 28134 consensus_queue.cc:1035] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:16.160923 28135 tablet_replica.cc:439] T d455497f3e9a4b4188be5af8627cd54f P ace0868b811e408ea26ba92547cf51ec: TxnStatusTablet state changed. Reason: New leader 269ac13e0d0343a894d0bb87a2ea673c. Latest consensus state: current_term: 1 leader_uuid: "269ac13e0d0343a894d0bb87a2ea673c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } } }
I20260928 16:39:16.161095 28135 tablet_replica.cc:442] T d455497f3e9a4b4188be5af8627cd54f P ace0868b811e408ea26ba92547cf51ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:39:16.161670 28133 tablet_replica.cc:439] T d455497f3e9a4b4188be5af8627cd54f P 2a30debbc5e345fa8095f643137028a0: TxnStatusTablet state changed. Reason: New leader 269ac13e0d0343a894d0bb87a2ea673c. Latest consensus state: current_term: 1 leader_uuid: "269ac13e0d0343a894d0bb87a2ea673c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } } }
I20260928 16:39:16.161875 28133 tablet_replica.cc:442] T d455497f3e9a4b4188be5af8627cd54f P 2a30debbc5e345fa8095f643137028a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:39:16.165186 28149 tablet_replica.cc:439] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "269ac13e0d0343a894d0bb87a2ea673c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } } }
I20260928 16:39:16.165374 28149 tablet_replica.cc:442] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:39:16.165573 28135 tablet_replica.cc:439] T d455497f3e9a4b4188be5af8627cd54f P ace0868b811e408ea26ba92547cf51ec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "269ac13e0d0343a894d0bb87a2ea673c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } } }
I20260928 16:39:16.165678 28135 tablet_replica.cc:442] T d455497f3e9a4b4188be5af8627cd54f P ace0868b811e408ea26ba92547cf51ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:39:16.167697 28155 mvcc.cc:205] Tried to move back new op lower bound from 7334353126005870592 to 7334353125910556672. Current Snapshot: MvccSnapshot[applied={T|T < 7334353126005870592}]
I20260928 16:39:16.171550 28134 tablet_replica.cc:439] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "269ac13e0d0343a894d0bb87a2ea673c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } } }
I20260928 16:39:16.171716 28134 tablet_replica.cc:442] T d455497f3e9a4b4188be5af8627cd54f P 269ac13e0d0343a894d0bb87a2ea673c: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:39:16.174656 28156 mvcc.cc:205] Tried to move back new op lower bound from 7334353126005870592 to 7334353125910556672. Current Snapshot: MvccSnapshot[applied={T|T < 7334353126005870592}]
I20260928 16:39:16.174928 28133 tablet_replica.cc:439] T d455497f3e9a4b4188be5af8627cd54f P 2a30debbc5e345fa8095f643137028a0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "269ac13e0d0343a894d0bb87a2ea673c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } } }
I20260928 16:39:16.175065 28133 tablet_replica.cc:442] T d455497f3e9a4b4188be5af8627cd54f P 2a30debbc5e345fa8095f643137028a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:39:16.183450 28125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:16.177683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45872 (local address 127.25.71.66:35315)
0928 16:39:16.178163 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:16.178222 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:16.178262 (+ 40us) server_negotiation.cc:407] Connection header received
0928 16:39:16.178377 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:16.178380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:16.178458 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:16.178575 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:16.179005 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.179875 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.181119 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.181492 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.181614 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:16.182025 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:16.182919 (+ 894us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:16.183058 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:16.183165 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 16:39:16.183255 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364}
I20260928 16:39:16.200116 28122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:16.196115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:35219 (local address 127.25.71.67:41515)
0928 16:39:16.196259 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:16.196264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:16.196515 (+ 251us) server_negotiation.cc:407] Connection header received
0928 16:39:16.196671 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:16.196674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:16.196721 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:16.196786 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:16.197066 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.197586 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.198494 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.198768 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.198976 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:16.199615 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:16.199701 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:16.199780 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:16.199900 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 16:39:16.199976 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:39:16.200224 28166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:16.195963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:41515 (local address 127.25.71.66:35219)
0928 16:39:16.196400 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:16.196415 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:16.196564 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:16.196812 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:16.196816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:16.196823 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:39:16.196958 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:16.196964 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.197702 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.197705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:16.198372 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:16.198382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.199064 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.199070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:16.199223 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:16.199311 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:39:16.199796 (+ 485us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:39:16.199804 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:39:16.199881 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 16:39:16.200004 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":142,"threads_started":1}
I20260928 16:39:16.210570 28046 raft_consensus.cc:1261] T f6883603e4674f218eb3eedeae94e16b P 269ac13e0d0343a894d0bb87a2ea673c [term 1 FOLLOWER]: Refusing update from remote peer ace0868b811e408ea26ba92547cf51ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:16.211102 27775 raft_consensus.cc:1261] T f6883603e4674f218eb3eedeae94e16b P 2a30debbc5e345fa8095f643137028a0 [term 1 FOLLOWER]: Refusing update from remote peer ace0868b811e408ea26ba92547cf51ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:16.211361 28135 consensus_queue.cc:1035] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:16.211695 28137 consensus_queue.cc:1035] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:16.217710 28154 mvcc.cc:205] Tried to move back new op lower bound from 7334353126231359488 to 7334353125118070784. Current Snapshot: MvccSnapshot[applied={T|T < 7334353126231359488}]
I20260928 16:39:16.242959 28110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:16.237777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:35315 (local address 127.25.71.67:43189)
0928 16:39:16.238077 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:16.238097 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:16.238222 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:16.238550 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:16.238553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:16.238560 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:39:16.238702 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:16.238708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.239753 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.239759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:16.240919 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:16.240942 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.241847 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.241852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:16.241977 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:16.242061 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:39:16.242482 (+ 421us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:39:16.242490 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:39:16.242560 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 16:39:16.242739 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260928 16:39:16.242987 28125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:16.237815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:43189 (local address 127.25.71.66:35315)
0928 16:39:16.238096 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:16.238102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:16.238324 (+ 222us) server_negotiation.cc:407] Connection header received
0928 16:39:16.238387 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:16.238390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:16.238456 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:16.238543 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:16.238837 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.239522 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.241135 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:16.241446 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:16.241806 (+ 360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:16.242240 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:16.242368 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:16.242452 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:16.242593 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 16:39:16.242695 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260928 16:39:16.282261 27616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:16.282581 27616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:16.287446 27616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37: Sending DeleteTablet for 3 replicas of tablet f6883603e4674f218eb3eedeae94e16b
I20260928 16:39:16.289590 28037 tablet_service.cc:1514] Processing DeleteTablet for tablet f6883603e4674f218eb3eedeae94e16b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:16 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:45126
I20260928 16:39:16.291280 27896 tablet_service.cc:1514] Processing DeleteTablet for tablet f6883603e4674f218eb3eedeae94e16b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:16 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:45868
I20260928 16:39:16.292627 27755 tablet_service.cc:1514] Processing DeleteTablet for tablet f6883603e4674f218eb3eedeae94e16b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:16 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:47390
I20260928 16:39:16.295047 27616 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:39:16.295789 27616 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:39:16.298952 28173 tablet_replica.cc:330] T f6883603e4674f218eb3eedeae94e16b P 269ac13e0d0343a894d0bb87a2ea673c: stopping tablet replica
I20260928 16:39:16.299405 28173 raft_consensus.cc:2229] T f6883603e4674f218eb3eedeae94e16b P 269ac13e0d0343a894d0bb87a2ea673c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:16.299625 28173 raft_consensus.cc:2258] T f6883603e4674f218eb3eedeae94e16b P 269ac13e0d0343a894d0bb87a2ea673c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:16.303315 28173 ts_tablet_manager.cc:1905] T f6883603e4674f218eb3eedeae94e16b P 269ac13e0d0343a894d0bb87a2ea673c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:16.303967 27755 tablet_service.cc:1467] Processing CreateTablet for tablet b7424de5149d499292106f51b11822fb (DEFAULT_TABLE table=test-table [id=594bdc946c964b07b3383fca36abfc75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:16.304419 27755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7424de5149d499292106f51b11822fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:16.307292 28037 tablet_service.cc:1467] Processing CreateTablet for tablet b7424de5149d499292106f51b11822fb (DEFAULT_TABLE table=test-table [id=594bdc946c964b07b3383fca36abfc75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:16.307662 28037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7424de5149d499292106f51b11822fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:16.309806 28173 ts_tablet_manager.cc:1918] T f6883603e4674f218eb3eedeae94e16b P 269ac13e0d0343a894d0bb87a2ea673c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:39:16.309954 28173 log.cc:1196] T f6883603e4674f218eb3eedeae94e16b P 269ac13e0d0343a894d0bb87a2ea673c: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/f6883603e4674f218eb3eedeae94e16b
I20260928 16:39:16.310323 28173 ts_tablet_manager.cc:1939] T f6883603e4674f218eb3eedeae94e16b P 269ac13e0d0343a894d0bb87a2ea673c: Deleting consensus metadata
I20260928 16:39:16.311518 27896 tablet_service.cc:1467] Processing CreateTablet for tablet b7424de5149d499292106f51b11822fb (DEFAULT_TABLE table=test-table [id=594bdc946c964b07b3383fca36abfc75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:16.311960 27896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7424de5149d499292106f51b11822fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:16.311962 27602 catalog_manager.cc:4841] TS 269ac13e0d0343a894d0bb87a2ea673c (127.25.71.67:41515): tablet f6883603e4674f218eb3eedeae94e16b (table test-table [id=eb9ad4cc7b84448ea80e594f7cc03a4c]) successfully deleted
I20260928 16:39:16.313910 28128 tablet_bootstrap.cc:492] T b7424de5149d499292106f51b11822fb P 2a30debbc5e345fa8095f643137028a0: Bootstrap starting.
I20260928 16:39:16.314131 28127 tablet_bootstrap.cc:492] T b7424de5149d499292106f51b11822fb P 269ac13e0d0343a894d0bb87a2ea673c: Bootstrap starting.
I20260928 16:39:16.314755 28175 tablet_replica.cc:330] T f6883603e4674f218eb3eedeae94e16b P 2a30debbc5e345fa8095f643137028a0: stopping tablet replica
I20260928 16:39:16.314990 28128 tablet_bootstrap.cc:654] T b7424de5149d499292106f51b11822fb P 2a30debbc5e345fa8095f643137028a0: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:16.315117 28175 raft_consensus.cc:2229] T f6883603e4674f218eb3eedeae94e16b P 2a30debbc5e345fa8095f643137028a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:16.315373 28175 raft_consensus.cc:2258] T f6883603e4674f218eb3eedeae94e16b P 2a30debbc5e345fa8095f643137028a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:16.315408 28127 tablet_bootstrap.cc:654] T b7424de5149d499292106f51b11822fb P 269ac13e0d0343a894d0bb87a2ea673c: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:16.315745 28174 tablet_replica.cc:330] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec: stopping tablet replica
I20260928 16:39:16.316068 28174 raft_consensus.cc:2229] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:39:16.316473 28174 raft_consensus.cc:2258] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:16.317766 28175 ts_tablet_manager.cc:1905] T f6883603e4674f218eb3eedeae94e16b P 2a30debbc5e345fa8095f643137028a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:16.319754 28131 tablet_bootstrap.cc:492] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec: Bootstrap starting.
I20260928 16:39:16.321053 28131 tablet_bootstrap.cc:654] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:16.322001 28175 ts_tablet_manager.cc:1918] T f6883603e4674f218eb3eedeae94e16b P 2a30debbc5e345fa8095f643137028a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:39:16.322139 28175 log.cc:1196] T f6883603e4674f218eb3eedeae94e16b P 2a30debbc5e345fa8095f643137028a0: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-0/wal/wals/f6883603e4674f218eb3eedeae94e16b
I20260928 16:39:16.322499 28175 ts_tablet_manager.cc:1939] T f6883603e4674f218eb3eedeae94e16b P 2a30debbc5e345fa8095f643137028a0: Deleting consensus metadata
I20260928 16:39:16.323822 28127 tablet_bootstrap.cc:492] T b7424de5149d499292106f51b11822fb P 269ac13e0d0343a894d0bb87a2ea673c: No bootstrap required, opened a new log
I20260928 16:39:16.323971 28127 ts_tablet_manager.cc:1397] T b7424de5149d499292106f51b11822fb P 269ac13e0d0343a894d0bb87a2ea673c: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20260928 16:39:16.324411 27603 catalog_manager.cc:4841] TS 2a30debbc5e345fa8095f643137028a0 (127.25.71.65:33657): tablet f6883603e4674f218eb3eedeae94e16b (table test-table [id=eb9ad4cc7b84448ea80e594f7cc03a4c]) successfully deleted
I20260928 16:39:16.324612 28127 raft_consensus.cc:348] T b7424de5149d499292106f51b11822fb P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:16.324777 28127 raft_consensus.cc:374] T b7424de5149d499292106f51b11822fb P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:16.324833 28127 raft_consensus.cc:729] T b7424de5149d499292106f51b11822fb P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269ac13e0d0343a894d0bb87a2ea673c, State: Initialized, Role: FOLLOWER
I20260928 16:39:16.325381 28127 consensus_queue.cc:260] T b7424de5149d499292106f51b11822fb P 269ac13e0d0343a894d0bb87a2ea673c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:16.325762 28127 ts_tablet_manager.cc:1428] T b7424de5149d499292106f51b11822fb P 269ac13e0d0343a894d0bb87a2ea673c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 16:39:16.327041 28174 ts_tablet_manager.cc:1905] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:16.330407 28131 tablet_bootstrap.cc:492] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec: No bootstrap required, opened a new log
I20260928 16:39:16.330564 28131 ts_tablet_manager.cc:1397] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20260928 16:39:16.331240 28131 raft_consensus.cc:348] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:16.331411 28131 raft_consensus.cc:374] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:16.331456 28131 raft_consensus.cc:729] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ace0868b811e408ea26ba92547cf51ec, State: Initialized, Role: FOLLOWER
I20260928 16:39:16.331655 28131 consensus_queue.cc:260] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:16.332005 28131 ts_tablet_manager.cc:1428] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:39:16.335072 28174 ts_tablet_manager.cc:1918] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:39:16.335273 28174 log.cc:1196] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/f6883603e4674f218eb3eedeae94e16b
I20260928 16:39:16.335675 28174 ts_tablet_manager.cc:1939] T f6883603e4674f218eb3eedeae94e16b P ace0868b811e408ea26ba92547cf51ec: Deleting consensus metadata
I20260928 16:39:16.337508 27602 catalog_manager.cc:4841] TS ace0868b811e408ea26ba92547cf51ec (127.25.71.66:35315): tablet f6883603e4674f218eb3eedeae94e16b (table test-table [id=eb9ad4cc7b84448ea80e594f7cc03a4c]) successfully deleted
I20260928 16:39:16.338646 28128 tablet_bootstrap.cc:492] T b7424de5149d499292106f51b11822fb P 2a30debbc5e345fa8095f643137028a0: No bootstrap required, opened a new log
I20260928 16:39:16.338792 28128 ts_tablet_manager.cc:1397] T b7424de5149d499292106f51b11822fb P 2a30debbc5e345fa8095f643137028a0: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20260928 16:39:16.339358 28128 raft_consensus.cc:348] T b7424de5149d499292106f51b11822fb P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:16.339537 28128 raft_consensus.cc:374] T b7424de5149d499292106f51b11822fb P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:16.339573 28128 raft_consensus.cc:729] T b7424de5149d499292106f51b11822fb P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a30debbc5e345fa8095f643137028a0, State: Initialized, Role: FOLLOWER
I20260928 16:39:16.339740 28128 consensus_queue.cc:260] T b7424de5149d499292106f51b11822fb P 2a30debbc5e345fa8095f643137028a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:16.340063 28128 ts_tablet_manager.cc:1428] T b7424de5149d499292106f51b11822fb P 2a30debbc5e345fa8095f643137028a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:39:16.353104 28137 raft_consensus.cc:482] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:39:16.353327 28137 raft_consensus.cc:504] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:16.354030 28137 leader_election.cc:288] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a30debbc5e345fa8095f643137028a0 (127.25.71.65:33657), 269ac13e0d0343a894d0bb87a2ea673c (127.25.71.67:41515)
I20260928 16:39:16.354524 28046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7424de5149d499292106f51b11822fb" candidate_uuid: "ace0868b811e408ea26ba92547cf51ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269ac13e0d0343a894d0bb87a2ea673c" is_pre_election: true
I20260928 16:39:16.354612 27775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7424de5149d499292106f51b11822fb" candidate_uuid: "ace0868b811e408ea26ba92547cf51ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a30debbc5e345fa8095f643137028a0" is_pre_election: true
I20260928 16:39:16.354768 28046 raft_consensus.cc:2454] T b7424de5149d499292106f51b11822fb P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ace0868b811e408ea26ba92547cf51ec in term 0.
I20260928 16:39:16.354777 27775 raft_consensus.cc:2454] T b7424de5149d499292106f51b11822fb P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ace0868b811e408ea26ba92547cf51ec in term 0.
I20260928 16:39:16.355120 27849 leader_election.cc:302] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269ac13e0d0343a894d0bb87a2ea673c, ace0868b811e408ea26ba92547cf51ec; no voters:
I20260928 16:39:16.355347 28137 raft_consensus.cc:2781] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:39:16.355427 28137 raft_consensus.cc:482] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:39:16.355464 28137 raft_consensus.cc:3037] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:16.357045 28137 raft_consensus.cc:504] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:16.357542 28137 leader_election.cc:288] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec [CANDIDATE]: Term 1 election: Requested vote from peers 2a30debbc5e345fa8095f643137028a0 (127.25.71.65:33657), 269ac13e0d0343a894d0bb87a2ea673c (127.25.71.67:41515)
I20260928 16:39:16.358108 28046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7424de5149d499292106f51b11822fb" candidate_uuid: "ace0868b811e408ea26ba92547cf51ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269ac13e0d0343a894d0bb87a2ea673c"
I20260928 16:39:16.358144 27775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7424de5149d499292106f51b11822fb" candidate_uuid: "ace0868b811e408ea26ba92547cf51ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a30debbc5e345fa8095f643137028a0"
I20260928 16:39:16.358284 28046 raft_consensus.cc:3037] T b7424de5149d499292106f51b11822fb P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:16.358311 27775 raft_consensus.cc:3037] T b7424de5149d499292106f51b11822fb P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:16.359586 28046 raft_consensus.cc:2454] T b7424de5149d499292106f51b11822fb P 269ac13e0d0343a894d0bb87a2ea673c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ace0868b811e408ea26ba92547cf51ec in term 1.
I20260928 16:39:16.359767 27775 raft_consensus.cc:2454] T b7424de5149d499292106f51b11822fb P 2a30debbc5e345fa8095f643137028a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ace0868b811e408ea26ba92547cf51ec in term 1.
I20260928 16:39:16.359954 27849 leader_election.cc:302] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269ac13e0d0343a894d0bb87a2ea673c, ace0868b811e408ea26ba92547cf51ec; no voters:
I20260928 16:39:16.360219 28137 raft_consensus.cc:2781] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:16.360396 28137 raft_consensus.cc:686] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec [term 1 LEADER]: Becoming Leader. State: Replica: ace0868b811e408ea26ba92547cf51ec, State: Running, Role: LEADER
I20260928 16:39:16.360584 28137 consensus_queue.cc:237] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:16.362913 27616 catalog_manager.cc:5495] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec reported cstate change: term changed from 0 to 1, leader changed from <none> to ace0868b811e408ea26ba92547cf51ec (127.25.71.66). New cstate: current_term: 1 leader_uuid: "ace0868b811e408ea26ba92547cf51ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } health_report { overall_health: UNKNOWN } } }
I20260928 16:39:16.389151 28046 raft_consensus.cc:1261] T b7424de5149d499292106f51b11822fb P 269ac13e0d0343a894d0bb87a2ea673c [term 1 FOLLOWER]: Refusing update from remote peer ace0868b811e408ea26ba92547cf51ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:16.389469 27775 raft_consensus.cc:1261] T b7424de5149d499292106f51b11822fb P 2a30debbc5e345fa8095f643137028a0 [term 1 FOLLOWER]: Refusing update from remote peer ace0868b811e408ea26ba92547cf51ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:16.389683 28137 consensus_queue.cc:1035] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:16.390251 28137 consensus_queue.cc:1035] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 16:39:16.408093 28155 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 16:39:17.978382 27616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:17.978690 27616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:17.982750 27616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37: Sending DeleteTablet for 3 replicas of tablet b7424de5149d499292106f51b11822fb
I20260928 16:39:17.983646 27755 tablet_service.cc:1514] Processing DeleteTablet for tablet b7424de5149d499292106f51b11822fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:17 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:47390
I20260928 16:39:17.983717 28037 tablet_service.cc:1514] Processing DeleteTablet for tablet b7424de5149d499292106f51b11822fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:17 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:45126
I20260928 16:39:17.984118 27896 tablet_service.cc:1514] Processing DeleteTablet for tablet b7424de5149d499292106f51b11822fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:17 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:45868
I20260928 16:39:17.984622 28205 tablet_replica.cc:330] T b7424de5149d499292106f51b11822fb P 269ac13e0d0343a894d0bb87a2ea673c: stopping tablet replica
I20260928 16:39:17.984799 28205 raft_consensus.cc:2229] T b7424de5149d499292106f51b11822fb P 269ac13e0d0343a894d0bb87a2ea673c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:17.985020 28205 raft_consensus.cc:2258] T b7424de5149d499292106f51b11822fb P 269ac13e0d0343a894d0bb87a2ea673c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:17.985342 28206 tablet_replica.cc:330] T b7424de5149d499292106f51b11822fb P 2a30debbc5e345fa8095f643137028a0: stopping tablet replica
I20260928 16:39:17.985507 28206 raft_consensus.cc:2229] T b7424de5149d499292106f51b11822fb P 2a30debbc5e345fa8095f643137028a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:17.985663 28206 raft_consensus.cc:2258] T b7424de5149d499292106f51b11822fb P 2a30debbc5e345fa8095f643137028a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:17.986119 28205 ts_tablet_manager.cc:1905] T b7424de5149d499292106f51b11822fb P 269ac13e0d0343a894d0bb87a2ea673c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:17.986464 28206 ts_tablet_manager.cc:1905] T b7424de5149d499292106f51b11822fb P 2a30debbc5e345fa8095f643137028a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:17.987775 28207 tablet_replica.cc:330] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec: stopping tablet replica
I20260928 16:39:17.987946 28207 raft_consensus.cc:2229] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:39:17.988245 28207 raft_consensus.cc:2258] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:17.989182 28207 ts_tablet_manager.cc:1905] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:17.989678 28205 ts_tablet_manager.cc:1918] T b7424de5149d499292106f51b11822fb P 269ac13e0d0343a894d0bb87a2ea673c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:39:17.989678 28206 ts_tablet_manager.cc:1918] T b7424de5149d499292106f51b11822fb P 2a30debbc5e345fa8095f643137028a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:39:17.989782 28205 log.cc:1196] T b7424de5149d499292106f51b11822fb P 269ac13e0d0343a894d0bb87a2ea673c: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/b7424de5149d499292106f51b11822fb
I20260928 16:39:17.989784 28206 log.cc:1196] T b7424de5149d499292106f51b11822fb P 2a30debbc5e345fa8095f643137028a0: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-0/wal/wals/b7424de5149d499292106f51b11822fb
I20260928 16:39:17.990077 28206 ts_tablet_manager.cc:1939] T b7424de5149d499292106f51b11822fb P 2a30debbc5e345fa8095f643137028a0: Deleting consensus metadata
I20260928 16:39:17.990077 28205 ts_tablet_manager.cc:1939] T b7424de5149d499292106f51b11822fb P 269ac13e0d0343a894d0bb87a2ea673c: Deleting consensus metadata
I20260928 16:39:17.991458 27602 catalog_manager.cc:4841] TS 269ac13e0d0343a894d0bb87a2ea673c (127.25.71.67:41515): tablet b7424de5149d499292106f51b11822fb (table test-table [id=594bdc946c964b07b3383fca36abfc75]) successfully deleted
I20260928 16:39:17.991458 27603 catalog_manager.cc:4841] TS 2a30debbc5e345fa8095f643137028a0 (127.25.71.65:33657): tablet b7424de5149d499292106f51b11822fb (table test-table [id=594bdc946c964b07b3383fca36abfc75]) successfully deleted
I20260928 16:39:17.992894 28207 ts_tablet_manager.cc:1918] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:39:17.992996 28207 log.cc:1196] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/b7424de5149d499292106f51b11822fb
I20260928 16:39:17.993300 28207 ts_tablet_manager.cc:1939] T b7424de5149d499292106f51b11822fb P ace0868b811e408ea26ba92547cf51ec: Deleting consensus metadata
I20260928 16:39:17.994208 27602 catalog_manager.cc:4841] TS ace0868b811e408ea26ba92547cf51ec (127.25.71.66:35315): tablet b7424de5149d499292106f51b11822fb (table test-table [id=594bdc946c964b07b3383fca36abfc75]) successfully deleted
Sep 28 16:39:17 dist-test-slave-nqgh krb5kdc[27571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613557, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:39:18 dist-test-slave-nqgh krb5kdc[27571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613557, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:18.028870 28215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:18.009325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42332 (local address 127.25.71.126:40891)
0928 16:39:18.009661 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:18.009669 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:18.009825 (+ 156us) server_negotiation.cc:407] Connection header received
0928 16:39:18.010056 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:18.010058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:18.010116 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:18.010291 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:18.011339 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:18.012021 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:18.013064 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:18.013358 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:18.013462 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:18.024327 (+ 10865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:18.024333 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:18.024336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:18.024401 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:18.026219 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:18.026990 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:18.026994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:18.026995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:18.027123 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:18.027540 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:18.027544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:18.027546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:18.027827 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:18.027973 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:18.028399 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 16:39:18.028637 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":127,"threads_started":1}
I20260928 16:39:18.032850 27616 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:39:18.033406 27616 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:39:18.040223 27896 tablet_service.cc:1467] Processing CreateTablet for tablet 726d78b0e13b49d78049a286cfdc9f36 (DEFAULT_TABLE table=test-table [id=17344977665e440e9340decf0ffea93b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:18.040225 28037 tablet_service.cc:1467] Processing CreateTablet for tablet 726d78b0e13b49d78049a286cfdc9f36 (DEFAULT_TABLE table=test-table [id=17344977665e440e9340decf0ffea93b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:18.040380 27755 tablet_service.cc:1467] Processing CreateTablet for tablet 726d78b0e13b49d78049a286cfdc9f36 (DEFAULT_TABLE table=test-table [id=17344977665e440e9340decf0ffea93b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:18.040644 28037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 726d78b0e13b49d78049a286cfdc9f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:18.040644 27896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 726d78b0e13b49d78049a286cfdc9f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:18.040762 27755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 726d78b0e13b49d78049a286cfdc9f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:18.043152 28216 tablet_bootstrap.cc:492] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0: Bootstrap starting.
I20260928 16:39:18.044023 28218 tablet_bootstrap.cc:492] T 726d78b0e13b49d78049a286cfdc9f36 P ace0868b811e408ea26ba92547cf51ec: Bootstrap starting.
I20260928 16:39:18.044085 28217 tablet_bootstrap.cc:492] T 726d78b0e13b49d78049a286cfdc9f36 P 269ac13e0d0343a894d0bb87a2ea673c: Bootstrap starting.
I20260928 16:39:18.044301 28216 tablet_bootstrap.cc:654] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:18.045074 28217 tablet_bootstrap.cc:654] T 726d78b0e13b49d78049a286cfdc9f36 P 269ac13e0d0343a894d0bb87a2ea673c: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:18.045089 28218 tablet_bootstrap.cc:654] T 726d78b0e13b49d78049a286cfdc9f36 P ace0868b811e408ea26ba92547cf51ec: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:18.046247 28216 tablet_bootstrap.cc:492] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0: No bootstrap required, opened a new log
I20260928 16:39:18.046368 28217 tablet_bootstrap.cc:492] T 726d78b0e13b49d78049a286cfdc9f36 P 269ac13e0d0343a894d0bb87a2ea673c: No bootstrap required, opened a new log
I20260928 16:39:18.046401 28218 tablet_bootstrap.cc:492] T 726d78b0e13b49d78049a286cfdc9f36 P ace0868b811e408ea26ba92547cf51ec: No bootstrap required, opened a new log
I20260928 16:39:18.046458 28217 ts_tablet_manager.cc:1397] T 726d78b0e13b49d78049a286cfdc9f36 P 269ac13e0d0343a894d0bb87a2ea673c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:39:18.046448 28216 ts_tablet_manager.cc:1397] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:39:18.046530 28218 ts_tablet_manager.cc:1397] T 726d78b0e13b49d78049a286cfdc9f36 P ace0868b811e408ea26ba92547cf51ec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:39:18.047019 28217 raft_consensus.cc:348] T 726d78b0e13b49d78049a286cfdc9f36 P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:18.047165 28217 raft_consensus.cc:374] T 726d78b0e13b49d78049a286cfdc9f36 P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:18.047215 28217 raft_consensus.cc:729] T 726d78b0e13b49d78049a286cfdc9f36 P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269ac13e0d0343a894d0bb87a2ea673c, State: Initialized, Role: FOLLOWER
I20260928 16:39:18.047106 28218 raft_consensus.cc:348] T 726d78b0e13b49d78049a286cfdc9f36 P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:18.047271 28218 raft_consensus.cc:374] T 726d78b0e13b49d78049a286cfdc9f36 P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:18.047310 28218 raft_consensus.cc:729] T 726d78b0e13b49d78049a286cfdc9f36 P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ace0868b811e408ea26ba92547cf51ec, State: Initialized, Role: FOLLOWER
I20260928 16:39:18.047472 28218 consensus_queue.cc:260] T 726d78b0e13b49d78049a286cfdc9f36 P ace0868b811e408ea26ba92547cf51ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:18.047487 28216 raft_consensus.cc:348] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:18.047405 28217 consensus_queue.cc:260] T 726d78b0e13b49d78049a286cfdc9f36 P 269ac13e0d0343a894d0bb87a2ea673c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:18.047583 28216 raft_consensus.cc:374] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:18.047762 28216 raft_consensus.cc:729] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a30debbc5e345fa8095f643137028a0, State: Initialized, Role: FOLLOWER
I20260928 16:39:18.047775 28218 ts_tablet_manager.cc:1428] T 726d78b0e13b49d78049a286cfdc9f36 P ace0868b811e408ea26ba92547cf51ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:39:18.047880 28217 ts_tablet_manager.cc:1428] T 726d78b0e13b49d78049a286cfdc9f36 P 269ac13e0d0343a894d0bb87a2ea673c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 16:39:18.047938 28216 consensus_queue.cc:260] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:18.048316 28216 ts_tablet_manager.cc:1428] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 16:39:18.068107 28222 raft_consensus.cc:482] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:39:18.068276 28222 raft_consensus.cc:504] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:18.069528 28222 leader_election.cc:288] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ace0868b811e408ea26ba92547cf51ec (127.25.71.66:35315), 269ac13e0d0343a894d0bb87a2ea673c (127.25.71.67:41515)
I20260928 16:39:18.075021 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:18.070291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:41515 (local address 127.25.71.65:39793)
0928 16:39:18.070735 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:18.070753 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:18.070897 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:18.071226 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:18.071229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:18.071253 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:18.071376 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:18.071382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:18.072261 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:18.072265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:18.073505 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:18.073517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:18.074482 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:18.074484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:18.074569 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:18.074595 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:39:18.074696 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 16:39:18.074854 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":28,"spinlock_wait_cycles":1920,"thread_start_us":91,"threads_started":1}
I20260928 16:39:18.075217 28226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:18.070426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:39793 (local address 127.25.71.67:41515)
0928 16:39:18.070735 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:18.070746 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:18.070910 (+ 164us) server_negotiation.cc:407] Connection header received
0928 16:39:18.070990 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:18.070993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:18.071073 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:18.071309 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:18.071478 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:18.072103 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:18.073735 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:18.074318 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:18.074478 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:18.074552 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:18.074776 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 16:39:18.074894 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":92,"threads_started":1}
I20260928 16:39:18.075592 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:18.069820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:35315 (local address 127.25.71.65:47889)
0928 16:39:18.070297 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:18.070320 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:18.070568 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:18.071167 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:18.071171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:18.071189 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:18.071368 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:18.071375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:18.072913 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:18.072917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:18.073948 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:18.073957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:18.075077 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:18.075080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:18.075323 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:18.075345 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 16:39:18.075408 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 16:39:18.075478 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":6,"thread_start_us":153,"threads_started":1}
I20260928 16:39:18.075740 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:18.070002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:47889 (local address 127.25.71.66:35315)
0928 16:39:18.070881 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:18.070892 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:18.070907 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:39:18.070967 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:18.070970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:18.071025 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:18.071131 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:18.072245 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:18.072768 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:18.074161 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:18.074929 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:18.075067 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:18.075123 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:18.075470 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 16:39:18.075573 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":155,"threads_started":1}
I20260928 16:39:18.075821 28046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726d78b0e13b49d78049a286cfdc9f36" candidate_uuid: "2a30debbc5e345fa8095f643137028a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269ac13e0d0343a894d0bb87a2ea673c" is_pre_election: true
I20260928 16:39:18.076007 28046 raft_consensus.cc:2454] T 726d78b0e13b49d78049a286cfdc9f36 P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a30debbc5e345fa8095f643137028a0 in term 0.
I20260928 16:39:18.076272 27915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726d78b0e13b49d78049a286cfdc9f36" candidate_uuid: "2a30debbc5e345fa8095f643137028a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ace0868b811e408ea26ba92547cf51ec" is_pre_election: true
I20260928 16:39:18.076470 27915 raft_consensus.cc:2454] T 726d78b0e13b49d78049a286cfdc9f36 P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a30debbc5e345fa8095f643137028a0 in term 0.
I20260928 16:39:18.076493 27708 leader_election.cc:302] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269ac13e0d0343a894d0bb87a2ea673c, 2a30debbc5e345fa8095f643137028a0; no voters:
I20260928 16:39:18.076853 28222 raft_consensus.cc:2781] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:39:18.076929 28222 raft_consensus.cc:482] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:39:18.076956 28222 raft_consensus.cc:3037] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:18.078526 28222 raft_consensus.cc:504] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:18.079077 28222 leader_election.cc:288] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 [CANDIDATE]: Term 1 election: Requested vote from peers ace0868b811e408ea26ba92547cf51ec (127.25.71.66:35315), 269ac13e0d0343a894d0bb87a2ea673c (127.25.71.67:41515)
I20260928 16:39:18.079798 27915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726d78b0e13b49d78049a286cfdc9f36" candidate_uuid: "2a30debbc5e345fa8095f643137028a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ace0868b811e408ea26ba92547cf51ec"
I20260928 16:39:18.080006 27915 raft_consensus.cc:3037] T 726d78b0e13b49d78049a286cfdc9f36 P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:18.080049 28046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726d78b0e13b49d78049a286cfdc9f36" candidate_uuid: "2a30debbc5e345fa8095f643137028a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269ac13e0d0343a894d0bb87a2ea673c"
I20260928 16:39:18.080250 28046 raft_consensus.cc:3037] T 726d78b0e13b49d78049a286cfdc9f36 P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:18.081427 28046 raft_consensus.cc:2454] T 726d78b0e13b49d78049a286cfdc9f36 P 269ac13e0d0343a894d0bb87a2ea673c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a30debbc5e345fa8095f643137028a0 in term 1.
I20260928 16:39:18.081579 27915 raft_consensus.cc:2454] T 726d78b0e13b49d78049a286cfdc9f36 P ace0868b811e408ea26ba92547cf51ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a30debbc5e345fa8095f643137028a0 in term 1.
I20260928 16:39:18.081913 27708 leader_election.cc:302] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269ac13e0d0343a894d0bb87a2ea673c, 2a30debbc5e345fa8095f643137028a0; no voters:
I20260928 16:39:18.082181 28222 raft_consensus.cc:2781] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:18.082480 28222 raft_consensus.cc:686] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 [term 1 LEADER]: Becoming Leader. State: Replica: 2a30debbc5e345fa8095f643137028a0, State: Running, Role: LEADER
I20260928 16:39:18.082800 28222 consensus_queue.cc:237] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:18.085985 27616 catalog_manager.cc:5495] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a30debbc5e345fa8095f643137028a0 (127.25.71.65). New cstate: current_term: 1 leader_uuid: "2a30debbc5e345fa8095f643137028a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } health_report { overall_health: UNKNOWN } } }
I20260928 16:39:18.118255 28231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:18.113174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.25.71.65:33657)
0928 16:39:18.113503 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:18.113511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:18.113526 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:39:18.113633 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:18.113635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:18.113707 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:18.113842 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:18.114323 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:18.114996 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:18.116092 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:18.116368 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:18.116458 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:18.116803 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:18.117547 (+ 744us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:18.117669 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:18.117846 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 16:39:18.117934 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":131,"threads_started":1}
I20260928 16:39:18.135643 28226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:18.128755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:36915 (local address 127.25.71.67:41515)
0928 16:39:18.128912 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:18.128916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:18.129970 (+ 1054us) server_negotiation.cc:407] Connection header received
0928 16:39:18.130398 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:18.130401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:18.130469 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:18.130697 (+ 228us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:18.131778 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:18.132935 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:18.134200 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:18.134455 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:18.134563 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:18.135048 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:18.135164 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:18.135229 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:18.135336 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 16:39:18.135430 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:39:18.135679 28234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:18.129024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:41515 (local address 127.25.71.65:36915)
0928 16:39:18.129805 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:18.129825 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:18.130039 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:18.130703 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:18.130707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:18.130718 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:39:18.131362 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:18.131372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:18.133133 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:18.133137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:18.133968 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:18.133984 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:18.134686 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:18.134691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:18.134785 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:18.134875 (+ 90us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:39:18.135248 (+ 373us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:39:18.135256 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:39:18.135319 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 16:39:18.135447 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":163,"threads_started":1}
I20260928 16:39:18.144527 28046 raft_consensus.cc:1261] T 726d78b0e13b49d78049a286cfdc9f36 P 269ac13e0d0343a894d0bb87a2ea673c [term 1 FOLLOWER]: Refusing update from remote peer 2a30debbc5e345fa8095f643137028a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:18.145128 27915 raft_consensus.cc:1261] T 726d78b0e13b49d78049a286cfdc9f36 P ace0868b811e408ea26ba92547cf51ec [term 1 FOLLOWER]: Refusing update from remote peer 2a30debbc5e345fa8095f643137028a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:18.145560 28222 consensus_queue.cc:1035] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:18.145845 28227 consensus_queue.cc:1035] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:18.170759 28231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:18.167124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:54805 (local address 127.25.71.65:33657)
0928 16:39:18.167261 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:18.167265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:18.167576 (+ 311us) server_negotiation.cc:407] Connection header received
0928 16:39:18.167700 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:18.167703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:18.167755 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:18.167821 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:18.168076 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:18.168624 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:18.169504 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:18.169740 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:18.169856 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:18.170202 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:18.170292 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:18.170373 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:18.170493 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 16:39:18.170554 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:39:18.170792 28239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:18.167105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:33657 (local address 127.25.71.67:54805)
0928 16:39:18.167481 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:18.167495 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:18.167607 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:18.167844 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:18.167853 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:18.167861 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:39:18.167984 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:18.167989 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:18.168741 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:18.168744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:18.169387 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:18.169396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:18.169881 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:18.169885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:18.170007 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:18.170052 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:39:18.170399 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:39:18.170408 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 16:39:18.170472 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 16:39:18.170586 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":174,"threads_started":1}
I20260928 16:39:18.202255 27616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:18.202488 27616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:18.205653 27616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37: Sending DeleteTablet for 3 replicas of tablet 726d78b0e13b49d78049a286cfdc9f36
I20260928 16:39:18.206614 27896 tablet_service.cc:1514] Processing DeleteTablet for tablet 726d78b0e13b49d78049a286cfdc9f36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:18 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:45868
I20260928 16:39:18.206782 28037 tablet_service.cc:1514] Processing DeleteTablet for tablet 726d78b0e13b49d78049a286cfdc9f36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:18 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:45126
I20260928 16:39:18.206930 28207 tablet_replica.cc:330] T 726d78b0e13b49d78049a286cfdc9f36 P ace0868b811e408ea26ba92547cf51ec: stopping tablet replica
I20260928 16:39:18.207109 28207 raft_consensus.cc:2229] T 726d78b0e13b49d78049a286cfdc9f36 P ace0868b811e408ea26ba92547cf51ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:18.207266 28207 raft_consensus.cc:2258] T 726d78b0e13b49d78049a286cfdc9f36 P ace0868b811e408ea26ba92547cf51ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:18.207418 28205 tablet_replica.cc:330] T 726d78b0e13b49d78049a286cfdc9f36 P 269ac13e0d0343a894d0bb87a2ea673c: stopping tablet replica
I20260928 16:39:18.207581 28205 raft_consensus.cc:2229] T 726d78b0e13b49d78049a286cfdc9f36 P 269ac13e0d0343a894d0bb87a2ea673c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:18.207741 27755 tablet_service.cc:1514] Processing DeleteTablet for tablet 726d78b0e13b49d78049a286cfdc9f36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:18 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:47390
I20260928 16:39:18.207790 28205 raft_consensus.cc:2258] T 726d78b0e13b49d78049a286cfdc9f36 P 269ac13e0d0343a894d0bb87a2ea673c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:18.208021 28206 tablet_replica.cc:330] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0: stopping tablet replica
I20260928 16:39:18.208170 28206 raft_consensus.cc:2229] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:39:18.208222 27616 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 16:39:18.208647 28206 raft_consensus.cc:2258] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 16:39:18.208768 27616 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:39:18.209290 28205 ts_tablet_manager.cc:1905] T 726d78b0e13b49d78049a286cfdc9f36 P 269ac13e0d0343a894d0bb87a2ea673c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:18.209444 28206 ts_tablet_manager.cc:1905] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:18.211623 28207 ts_tablet_manager.cc:1905] T 726d78b0e13b49d78049a286cfdc9f36 P ace0868b811e408ea26ba92547cf51ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:18.212728 28205 ts_tablet_manager.cc:1918] T 726d78b0e13b49d78049a286cfdc9f36 P 269ac13e0d0343a894d0bb87a2ea673c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:39:18.212826 28205 log.cc:1196] T 726d78b0e13b49d78049a286cfdc9f36 P 269ac13e0d0343a894d0bb87a2ea673c: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/726d78b0e13b49d78049a286cfdc9f36
I20260928 16:39:18.213093 28205 ts_tablet_manager.cc:1939] T 726d78b0e13b49d78049a286cfdc9f36 P 269ac13e0d0343a894d0bb87a2ea673c: Deleting consensus metadata
I20260928 16:39:18.213135 28206 ts_tablet_manager.cc:1918] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:39:18.213207 28206 log.cc:1196] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-0/wal/wals/726d78b0e13b49d78049a286cfdc9f36
I20260928 16:39:18.213434 28206 ts_tablet_manager.cc:1939] T 726d78b0e13b49d78049a286cfdc9f36 P 2a30debbc5e345fa8095f643137028a0: Deleting consensus metadata
I20260928 16:39:18.214361 27603 catalog_manager.cc:4841] TS 2a30debbc5e345fa8095f643137028a0 (127.25.71.65:33657): tablet 726d78b0e13b49d78049a286cfdc9f36 (table test-table [id=17344977665e440e9340decf0ffea93b]) successfully deleted
I20260928 16:39:18.215097 28207 ts_tablet_manager.cc:1918] T 726d78b0e13b49d78049a286cfdc9f36 P ace0868b811e408ea26ba92547cf51ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:39:18.215216 28207 log.cc:1196] T 726d78b0e13b49d78049a286cfdc9f36 P ace0868b811e408ea26ba92547cf51ec: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/726d78b0e13b49d78049a286cfdc9f36
I20260928 16:39:18.215216 27602 catalog_manager.cc:4841] TS 269ac13e0d0343a894d0bb87a2ea673c (127.25.71.67:41515): tablet 726d78b0e13b49d78049a286cfdc9f36 (table test-table [id=17344977665e440e9340decf0ffea93b]) successfully deleted
I20260928 16:39:18.215466 28207 ts_tablet_manager.cc:1939] T 726d78b0e13b49d78049a286cfdc9f36 P ace0868b811e408ea26ba92547cf51ec: Deleting consensus metadata
I20260928 16:39:18.216766 27602 catalog_manager.cc:4841] TS ace0868b811e408ea26ba92547cf51ec (127.25.71.66:35315): tablet 726d78b0e13b49d78049a286cfdc9f36 (table test-table [id=17344977665e440e9340decf0ffea93b]) successfully deleted
I20260928 16:39:18.218531 27755 tablet_service.cc:1467] Processing CreateTablet for tablet 3249f381d68c4ffd846252fe410c7390 (DEFAULT_TABLE table=test-table [id=45fc0dbdbda84f0789167204ff8121f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:18.218701 28037 tablet_service.cc:1467] Processing CreateTablet for tablet 3249f381d68c4ffd846252fe410c7390 (DEFAULT_TABLE table=test-table [id=45fc0dbdbda84f0789167204ff8121f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:18.218701 27896 tablet_service.cc:1467] Processing CreateTablet for tablet 3249f381d68c4ffd846252fe410c7390 (DEFAULT_TABLE table=test-table [id=45fc0dbdbda84f0789167204ff8121f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:18.218940 27755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3249f381d68c4ffd846252fe410c7390. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:18.219048 27896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3249f381d68c4ffd846252fe410c7390. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:18.219048 28037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3249f381d68c4ffd846252fe410c7390. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:18.221726 28216 tablet_bootstrap.cc:492] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0: Bootstrap starting.
I20260928 16:39:18.222396 28218 tablet_bootstrap.cc:492] T 3249f381d68c4ffd846252fe410c7390 P ace0868b811e408ea26ba92547cf51ec: Bootstrap starting.
I20260928 16:39:18.222949 28216 tablet_bootstrap.cc:654] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:18.223555 28218 tablet_bootstrap.cc:654] T 3249f381d68c4ffd846252fe410c7390 P ace0868b811e408ea26ba92547cf51ec: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:18.225116 28216 tablet_bootstrap.cc:492] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0: No bootstrap required, opened a new log
I20260928 16:39:18.225232 28216 ts_tablet_manager.cc:1397] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 16:39:18.225404 28218 tablet_bootstrap.cc:492] T 3249f381d68c4ffd846252fe410c7390 P ace0868b811e408ea26ba92547cf51ec: No bootstrap required, opened a new log
I20260928 16:39:18.225528 28218 ts_tablet_manager.cc:1397] T 3249f381d68c4ffd846252fe410c7390 P ace0868b811e408ea26ba92547cf51ec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:39:18.225769 28216 raft_consensus.cc:348] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:18.225930 28216 raft_consensus.cc:374] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:18.225962 28216 raft_consensus.cc:729] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a30debbc5e345fa8095f643137028a0, State: Initialized, Role: FOLLOWER
I20260928 16:39:18.226096 28216 consensus_queue.cc:260] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:18.226027 28218 raft_consensus.cc:348] T 3249f381d68c4ffd846252fe410c7390 P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:18.226193 28218 raft_consensus.cc:374] T 3249f381d68c4ffd846252fe410c7390 P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:18.226238 28218 raft_consensus.cc:729] T 3249f381d68c4ffd846252fe410c7390 P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ace0868b811e408ea26ba92547cf51ec, State: Initialized, Role: FOLLOWER
I20260928 16:39:18.226385 28216 ts_tablet_manager.cc:1428] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:39:18.226378 28218 consensus_queue.cc:260] T 3249f381d68c4ffd846252fe410c7390 P ace0868b811e408ea26ba92547cf51ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:18.226711 28218 ts_tablet_manager.cc:1428] T 3249f381d68c4ffd846252fe410c7390 P ace0868b811e408ea26ba92547cf51ec: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 16:39:18.226759 28217 tablet_bootstrap.cc:492] T 3249f381d68c4ffd846252fe410c7390 P 269ac13e0d0343a894d0bb87a2ea673c: Bootstrap starting.
I20260928 16:39:18.227819 28217 tablet_bootstrap.cc:654] T 3249f381d68c4ffd846252fe410c7390 P 269ac13e0d0343a894d0bb87a2ea673c: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:18.229555 28217 tablet_bootstrap.cc:492] T 3249f381d68c4ffd846252fe410c7390 P 269ac13e0d0343a894d0bb87a2ea673c: No bootstrap required, opened a new log
I20260928 16:39:18.229671 28217 ts_tablet_manager.cc:1397] T 3249f381d68c4ffd846252fe410c7390 P 269ac13e0d0343a894d0bb87a2ea673c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:39:18.230149 28217 raft_consensus.cc:348] T 3249f381d68c4ffd846252fe410c7390 P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:18.230296 28217 raft_consensus.cc:374] T 3249f381d68c4ffd846252fe410c7390 P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:18.230329 28217 raft_consensus.cc:729] T 3249f381d68c4ffd846252fe410c7390 P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269ac13e0d0343a894d0bb87a2ea673c, State: Initialized, Role: FOLLOWER
I20260928 16:39:18.230461 28217 consensus_queue.cc:260] T 3249f381d68c4ffd846252fe410c7390 P 269ac13e0d0343a894d0bb87a2ea673c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:18.230790 28217 ts_tablet_manager.cc:1428] T 3249f381d68c4ffd846252fe410c7390 P 269ac13e0d0343a894d0bb87a2ea673c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 16:39:18.317703 28240 raft_consensus.cc:482] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:39:18.317888 28240 raft_consensus.cc:504] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:18.318524 28240 leader_election.cc:288] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ace0868b811e408ea26ba92547cf51ec (127.25.71.66:35315), 269ac13e0d0343a894d0bb87a2ea673c (127.25.71.67:41515)
I20260928 16:39:18.319294 27915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3249f381d68c4ffd846252fe410c7390" candidate_uuid: "2a30debbc5e345fa8095f643137028a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ace0868b811e408ea26ba92547cf51ec" is_pre_election: true
I20260928 16:39:18.319445 28046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3249f381d68c4ffd846252fe410c7390" candidate_uuid: "2a30debbc5e345fa8095f643137028a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269ac13e0d0343a894d0bb87a2ea673c" is_pre_election: true
I20260928 16:39:18.319574 27915 raft_consensus.cc:2454] T 3249f381d68c4ffd846252fe410c7390 P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a30debbc5e345fa8095f643137028a0 in term 0.
I20260928 16:39:18.319617 28046 raft_consensus.cc:2454] T 3249f381d68c4ffd846252fe410c7390 P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a30debbc5e345fa8095f643137028a0 in term 0.
I20260928 16:39:18.320286 27708 leader_election.cc:302] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269ac13e0d0343a894d0bb87a2ea673c, 2a30debbc5e345fa8095f643137028a0; no voters:
I20260928 16:39:18.320581 28240 raft_consensus.cc:2781] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:39:18.320657 28240 raft_consensus.cc:482] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:39:18.320717 28240 raft_consensus.cc:3037] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:18.322190 28240 raft_consensus.cc:504] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:18.322777 28240 leader_election.cc:288] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 [CANDIDATE]: Term 1 election: Requested vote from peers ace0868b811e408ea26ba92547cf51ec (127.25.71.66:35315), 269ac13e0d0343a894d0bb87a2ea673c (127.25.71.67:41515)
I20260928 16:39:18.323339 27915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3249f381d68c4ffd846252fe410c7390" candidate_uuid: "2a30debbc5e345fa8095f643137028a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ace0868b811e408ea26ba92547cf51ec"
I20260928 16:39:18.323457 28046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3249f381d68c4ffd846252fe410c7390" candidate_uuid: "2a30debbc5e345fa8095f643137028a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269ac13e0d0343a894d0bb87a2ea673c"
I20260928 16:39:18.323534 27915 raft_consensus.cc:3037] T 3249f381d68c4ffd846252fe410c7390 P ace0868b811e408ea26ba92547cf51ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:18.323611 28046 raft_consensus.cc:3037] T 3249f381d68c4ffd846252fe410c7390 P 269ac13e0d0343a894d0bb87a2ea673c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:18.325014 28046 raft_consensus.cc:2454] T 3249f381d68c4ffd846252fe410c7390 P 269ac13e0d0343a894d0bb87a2ea673c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a30debbc5e345fa8095f643137028a0 in term 1.
I20260928 16:39:18.325016 27915 raft_consensus.cc:2454] T 3249f381d68c4ffd846252fe410c7390 P ace0868b811e408ea26ba92547cf51ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a30debbc5e345fa8095f643137028a0 in term 1.
I20260928 16:39:18.325536 27708 leader_election.cc:302] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a30debbc5e345fa8095f643137028a0, ace0868b811e408ea26ba92547cf51ec; no voters:
I20260928 16:39:18.325809 28240 raft_consensus.cc:2781] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:18.325987 28240 raft_consensus.cc:686] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 [term 1 LEADER]: Becoming Leader. State: Replica: 2a30debbc5e345fa8095f643137028a0, State: Running, Role: LEADER
I20260928 16:39:18.326432 28240 consensus_queue.cc:237] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } }
I20260928 16:39:18.329368 27616 catalog_manager.cc:5495] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a30debbc5e345fa8095f643137028a0 (127.25.71.65). New cstate: current_term: 1 leader_uuid: "2a30debbc5e345fa8095f643137028a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a30debbc5e345fa8095f643137028a0" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 } health_report { overall_health: UNKNOWN } } }
I20260928 16:39:18.376194 27915 raft_consensus.cc:1261] T 3249f381d68c4ffd846252fe410c7390 P ace0868b811e408ea26ba92547cf51ec [term 1 FOLLOWER]: Refusing update from remote peer 2a30debbc5e345fa8095f643137028a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:18.376212 28046 raft_consensus.cc:1261] T 3249f381d68c4ffd846252fe410c7390 P 269ac13e0d0343a894d0bb87a2ea673c [term 1 FOLLOWER]: Refusing update from remote peer 2a30debbc5e345fa8095f643137028a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:18.376730 28240 consensus_queue.cc:1035] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269ac13e0d0343a894d0bb87a2ea673c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:18.376926 28222 consensus_queue.cc:1035] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ace0868b811e408ea26ba92547cf51ec" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 16:39:18.396809 28186 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 16:39:19.974102 27616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:19.974285 27616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:19.977771 27616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5109fff1b9143c6817e5db99a9fbf37: Sending DeleteTablet for 3 replicas of tablet 3249f381d68c4ffd846252fe410c7390
I20260928 16:39:19.978612 27755 tablet_service.cc:1514] Processing DeleteTablet for tablet 3249f381d68c4ffd846252fe410c7390 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:19 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:47390
I20260928 16:39:19.978744 27896 tablet_service.cc:1514] Processing DeleteTablet for tablet 3249f381d68c4ffd846252fe410c7390 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:19 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:45868
I20260928 16:39:19.978786 28037 tablet_service.cc:1514] Processing DeleteTablet for tablet 3249f381d68c4ffd846252fe410c7390 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:19 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:45126
I20260928 16:39:19.979099 28269 tablet_replica.cc:330] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0: stopping tablet replica
I20260928 16:39:19.979249 28269 raft_consensus.cc:2229] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:39:19.979310 28270 tablet_replica.cc:330] T 3249f381d68c4ffd846252fe410c7390 P ace0868b811e408ea26ba92547cf51ec: stopping tablet replica
I20260928 16:39:19.979365 28271 tablet_replica.cc:330] T 3249f381d68c4ffd846252fe410c7390 P 269ac13e0d0343a894d0bb87a2ea673c: stopping tablet replica
I20260928 16:39:19.979485 28270 raft_consensus.cc:2229] T 3249f381d68c4ffd846252fe410c7390 P ace0868b811e408ea26ba92547cf51ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:19.979522 28269 raft_consensus.cc:2258] T 3249f381d68c4ffd846252fe410c7390 P 2a30debbc5e345fa8095f643137028a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:19.979558 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 27689
I20260928 16:39:19.979637 28270 raft_consensus.cc:2258] T 3249f381d68c4ffd846252fe410c7390 P ace0868b811e408ea26ba92547cf51ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:19.980787 28271 raft_consensus.cc:2229] T 3249f381d68c4ffd846252fe410c7390 P 269ac13e0d0343a894d0bb87a2ea673c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:19.980942 28271 raft_consensus.cc:2258] T 3249f381d68c4ffd846252fe410c7390 P 269ac13e0d0343a894d0bb87a2ea673c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:19.981189 28270 ts_tablet_manager.cc:1905] T 3249f381d68c4ffd846252fe410c7390 P ace0868b811e408ea26ba92547cf51ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:19.981932 28271 ts_tablet_manager.cc:1905] T 3249f381d68c4ffd846252fe410c7390 P 269ac13e0d0343a894d0bb87a2ea673c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:19.983969 28270 ts_tablet_manager.cc:1918] T 3249f381d68c4ffd846252fe410c7390 P ace0868b811e408ea26ba92547cf51ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:39:19.984086 28270 log.cc:1196] T 3249f381d68c4ffd846252fe410c7390 P ace0868b811e408ea26ba92547cf51ec: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/3249f381d68c4ffd846252fe410c7390
I20260928 16:39:19.984334 28270 ts_tablet_manager.cc:1939] T 3249f381d68c4ffd846252fe410c7390 P ace0868b811e408ea26ba92547cf51ec: Deleting consensus metadata
I20260928 16:39:19.985241 28271 ts_tablet_manager.cc:1918] T 3249f381d68c4ffd846252fe410c7390 P 269ac13e0d0343a894d0bb87a2ea673c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:39:19.985340 28271 log.cc:1196] T 3249f381d68c4ffd846252fe410c7390 P 269ac13e0d0343a894d0bb87a2ea673c: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/3249f381d68c4ffd846252fe410c7390
I20260928 16:39:19.985416 27602 catalog_manager.cc:4841] TS ace0868b811e408ea26ba92547cf51ec (127.25.71.66:35315): tablet 3249f381d68c4ffd846252fe410c7390 (table test-table [id=45fc0dbdbda84f0789167204ff8121f6]) successfully deleted
I20260928 16:39:19.985603 28271 ts_tablet_manager.cc:1939] T 3249f381d68c4ffd846252fe410c7390 P 269ac13e0d0343a894d0bb87a2ea673c: Deleting consensus metadata
I20260928 16:39:19.986441 27602 catalog_manager.cc:4841] TS 269ac13e0d0343a894d0bb87a2ea673c (127.25.71.67:41515): tablet 3249f381d68c4ffd846252fe410c7390 (table test-table [id=45fc0dbdbda84f0789167204ff8121f6]) successfully deleted
W20260928 16:39:19.990286 27603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.71.65:33657 (error 108)
I20260928 16:39:19.990496 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 27831
I20260928 16:39:19.993611 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:19.993063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:33657 (local address 127.0.0.1:47416)
0928 16:39:19.993389 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:19.993479 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.65:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":114,"threads_started":1}
W20260928 16:39:19.993824 27603 catalog_manager.cc:4584] TS 2a30debbc5e345fa8095f643137028a0 (127.25.71.65:33657): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3249f381d68c4ffd846252fe410c7390: Network error: Client connection negotiation failed: client connection to 127.25.71.65:33657: connect: Connection refused (error 111)
I20260928 16:39:19.998179 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 27972
I20260928 16:39:20.005260 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 27584
2026-09-28T16:39:20Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6392 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:39:20 dist-test-slave-nqgh krb5kdc[28276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:39:20 dist-test-slave-nqgh krb5kdc[28276](info): set up 2 sockets
Sep 28 16:39:20 dist-test-slave-nqgh krb5kdc[28276](info): commencing operation
krb5kdc: starting...
W20260928 16:39:20.988354 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:39:21 dist-test-slave-nqgh krb5kdc[28276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:39:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:39:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:39:21.132551 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:46403
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:39659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:46403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:21.264140 28289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:21.264458 28289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:21.264501 28289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:21.268733 28289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:21.268836 28289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:21.268855 28289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:21.268872 28289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:21.268885 28289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:39:21.274712 28289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:39659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:46403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:46403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:21.275915 28289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:21.276978 28289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:21.287299 28297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:21.287217 28294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:21.287262 28295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:21.287598 28289 server_base.cc:1034] running on GCE node
I20260928 16:39:21.288496 28289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:21.289819 28289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:21.291148 28289 hybrid_clock.cc:648] HybridClock initialized: now 1790613561291143 us; error 81 us; skew 500 ppm
Sep 28 16:39:21 dist-test-slave-nqgh krb5kdc[28276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613561, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:21.294744 28289 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:39:21.295945 28289 webserver.cc:460] Webserver started at http://127.25.71.126:33321/ using document root <none> and password file <none>
I20260928 16:39:21.296546 28289 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:21.296620 28289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:21.296826 28289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:21.299113 28289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "cbd4f98192ed4f16897de96703622b77"
format_stamp: "Formatted at 2026-09-28 16:39:21 on dist-test-slave-nqgh"
server_key: "24ba1555e81e2290c406ce04fb6c310d"
server_key_iv: "7478651b49faa64d0b32792acb29c76d"
server_key_version: "encryptionkey@0"
I20260928 16:39:21.299710 28289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "cbd4f98192ed4f16897de96703622b77"
format_stamp: "Formatted at 2026-09-28 16:39:21 on dist-test-slave-nqgh"
server_key: "24ba1555e81e2290c406ce04fb6c310d"
server_key_iv: "7478651b49faa64d0b32792acb29c76d"
server_key_version: "encryptionkey@0"
I20260928 16:39:21.304251 28289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:21.307458 28304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:21.309504 28289 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:21.309856 28289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "cbd4f98192ed4f16897de96703622b77"
format_stamp: "Formatted at 2026-09-28 16:39:21 on dist-test-slave-nqgh"
server_key: "24ba1555e81e2290c406ce04fb6c310d"
server_key_iv: "7478651b49faa64d0b32792acb29c76d"
server_key_version: "encryptionkey@0"
I20260928 16:39:21.310037 28289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:21.344592 28289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:21.348639 28289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:21.348863 28289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:21.357172 28289 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:46403
I20260928 16:39:21.357234 28356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:46403 every 8 connection(s)
I20260928 16:39:21.358270 28289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:39:21.361117 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 28289
I20260928 16:39:21.361238 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:39:21.361500 25885 external_mini_cluster.cc:1428] Setting key 0e903f7fc23408baee2ce42ed1461b27
I20260928 16:39:21.364509 28357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 16:39:21 dist-test-slave-nqgh krb5kdc[28276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:21.374433 28357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77: Bootstrap starting.
I20260928 16:39:21.378005 28357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:21.379149 28357 log.cc:824] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:21.381350 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:21.363879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40974 (local address 127.25.71.126:46403)
0928 16:39:21.364438 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:21.364461 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:21.364504 (+ 43us) server_negotiation.cc:407] Connection header received
0928 16:39:21.365271 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:21.365293 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:21.365710 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:21.366072 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:21.367481 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:21.368696 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:21.370261 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:21.371070 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:21.371245 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:21.375227 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:21.375235 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:21.375262 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:21.375317 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:21.377497 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:21.378242 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:21.378249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:21.378257 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:21.378421 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:21.378799 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:21.378801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:21.378803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:21.379385 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:21.379557 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:21.380076 (+ 519us) server_negotiation.cc:299] Negotiation successful
0928 16:39:21.380435 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":185,"threads_started":1}
I20260928 16:39:21.382114 28357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77: No bootstrap required, opened a new log
I20260928 16:39:21.387071 28357 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd4f98192ed4f16897de96703622b77" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 46403 } }
I20260928 16:39:21.387445 28357 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:21.387487 28357 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbd4f98192ed4f16897de96703622b77, State: Initialized, Role: FOLLOWER
I20260928 16:39:21.388109 28357 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd4f98192ed4f16897de96703622b77" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 46403 } }
I20260928 16:39:21.388319 28357 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:39:21.388402 28357 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:39:21.388527 28357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:21.389914 28357 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd4f98192ed4f16897de96703622b77" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 46403 } }
I20260928 16:39:21.390393 28357 leader_election.cc:302] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbd4f98192ed4f16897de96703622b77; no voters:
I20260928 16:39:21.390833 28357 leader_election.cc:288] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:39:21.391083 28362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:21.391518 28362 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77 [term 1 LEADER]: Becoming Leader. State: Replica: cbd4f98192ed4f16897de96703622b77, State: Running, Role: LEADER
I20260928 16:39:21.391984 28362 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd4f98192ed4f16897de96703622b77" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 46403 } }
I20260928 16:39:21.392554 28357 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:21.393988 28363 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbd4f98192ed4f16897de96703622b77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd4f98192ed4f16897de96703622b77" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 46403 } } }
I20260928 16:39:21.394181 28363 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:21.394755 28371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:39:21.394743 28364 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbd4f98192ed4f16897de96703622b77. Latest consensus state: current_term: 1 leader_uuid: "cbd4f98192ed4f16897de96703622b77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd4f98192ed4f16897de96703622b77" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 46403 } } }
I20260928 16:39:21.395766 28364 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:21.398506 28371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:39:21.406571 28371 catalog_manager.cc:1347] Generated new cluster ID: a36727fd65db4e20998983b4f1f66a37
I20260928 16:39:21.406734 28371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:39:21.428807 28371 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:39:21.430275 28371 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:39:21.441306 28371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbd4f98192ed4f16897de96703622b77: Generated new TSK 0
I20260928 16:39:21.442705 28371 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
I20260928 16:39:21.487059 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:46403
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:39659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:21.604720 28383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:21.605022 28383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:21.605083 28383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:21.609328 28383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:21.609416 28383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:21.609504 28383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:39:21.615665 28383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:39659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:21.616858 28383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:21.617939 28383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:21.629599 28389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:21.629629 28391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:21.629634 28388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:21.630023 28383 server_base.cc:1034] running on GCE node
I20260928 16:39:21.630481 28383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:21.631263 28383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:21.632498 28383 hybrid_clock.cc:648] HybridClock initialized: now 1790613561632475 us; error 52 us; skew 500 ppm
Sep 28 16:39:21 dist-test-slave-nqgh krb5kdc[28276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613561, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:21.637630 28383 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:39:21.639019 28383 webserver.cc:460] Webserver started at http://127.25.71.65:34389/ using document root <none> and password file <none>
I20260928 16:39:21.639653 28383 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:21.639726 28383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:21.639946 28383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:21.642361 28383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "c6f5dcb4b8a347b1934d2252587351b9"
format_stamp: "Formatted at 2026-09-28 16:39:21 on dist-test-slave-nqgh"
server_key: "77c8f72159e08d674541faec31b45e9e"
server_key_iv: "6664f58837c3dfcfaeaf4a6723ab21c8"
server_key_version: "encryptionkey@0"
I20260928 16:39:21.643095 28383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "c6f5dcb4b8a347b1934d2252587351b9"
format_stamp: "Formatted at 2026-09-28 16:39:21 on dist-test-slave-nqgh"
server_key: "77c8f72159e08d674541faec31b45e9e"
server_key_iv: "6664f58837c3dfcfaeaf4a6723ab21c8"
server_key_version: "encryptionkey@0"
I20260928 16:39:21.648216 28383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:39:21.651575 28398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:21.653523 28383 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:21.653721 28383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "c6f5dcb4b8a347b1934d2252587351b9"
format_stamp: "Formatted at 2026-09-28 16:39:21 on dist-test-slave-nqgh"
server_key: "77c8f72159e08d674541faec31b45e9e"
server_key_iv: "6664f58837c3dfcfaeaf4a6723ab21c8"
server_key_version: "encryptionkey@0"
I20260928 16:39:21.653908 28383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:21.669922 28383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:21.674134 28383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:21.674371 28383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:21.675184 28383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:21.676241 28383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:21.676295 28383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:21.676349 28383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:21.676379 28383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:21.688367 28383 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:36361
I20260928 16:39:21.688503 28511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:36361 every 8 connection(s)
I20260928 16:39:21.689648 28383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:39:21.696199 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 28383
I20260928 16:39:21.696362 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:39:21.696759 25885 external_mini_cluster.cc:1428] Setting key 5de2dd0b73caa74d6f6bd0c61b9e74b4
Sep 28 16:39:21 dist-test-slave-nqgh krb5kdc[28276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613561, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:21.708627 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:21.692217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:41707 (local address 127.25.71.126:46403)
0928 16:39:21.692457 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:21.692461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:21.693941 (+ 1480us) server_negotiation.cc:407] Connection header received
0928 16:39:21.695491 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:21.695495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:21.695591 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:21.695688 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:21.697339 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:21.698065 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:21.699731 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:21.700003 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:21.700104 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:21.703526 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:21.703530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:21.703534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:21.703576 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:21.705389 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:21.706300 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:21.706304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:21.706306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:21.706455 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:21.706997 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:21.707001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:21.707002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:21.707341 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:21.707436 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:21.708258 (+ 822us) server_negotiation.cc:299] Negotiation successful
0928 16:39:21.708444 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
WARNING: no policy specified for kudu/127.25.71.66@KRBTEST.COM; defaulting to no policy
I20260928 16:39:21.709779 28514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:21.692845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:46403 (local address 127.25.71.65:41707)
0928 16:39:21.693709 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:21.693807 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:21.695246 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:21.695915 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:21.695922 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:21.696353 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:21.697149 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:21.697170 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:21.698202 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:21.698206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:21.699591 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:21.699604 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:21.700333 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:21.700338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:21.700557 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:21.701196 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:21.701215 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:21.703251 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:21.705602 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:21.705612 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:21.705636 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:21.706079 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:21.706582 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:21.706586 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:21.706588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:21.706852 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:21.707469 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:21.707477 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:21.707887 (+ 410us) client_negotiation.cc:769] Sending connection context
0928 16:39:21.708133 (+ 246us) client_negotiation.cc:240] Negotiation successful
0928 16:39:21.708557 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":171,"threads_started":1}
I20260928 16:39:21.711831 28512 heartbeater.cc:347] Connected to a master server at 127.25.71.126:46403
I20260928 16:39:21.712148 28512 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:21.712770 28512 heartbeater.cc:511] Master 127.25.71.126:46403 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.66@KRBTEST.COM" created.
I20260928 16:39:21.714931 28321 ts_manager.cc:194] Registered new tserver with Master: c6f5dcb4b8a347b1934d2252587351b9 (127.25.71.65:36361)
I20260928 16:39:21.716284 28321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:41707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
I20260928 16:39:21.736029 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:46403
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:39659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:21.876559 28517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:21.876981 28517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:21.877034 28517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:21.881801 28517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:21.881913 28517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:21.881984 28517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:39:21.888518 28517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:39659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:21.890035 28517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:21.891549 28517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:21.904624 28522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:21.904624 28523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:21.904773 28517 server_base.cc:1034] running on GCE node
W20260928 16:39:21.904641 28525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:21.905455 28517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:21.906322 28517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:21.907557 28517 hybrid_clock.cc:648] HybridClock initialized: now 1790613561907506 us; error 62 us; skew 500 ppm
Sep 28 16:39:21 dist-test-slave-nqgh krb5kdc[28276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613561, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:21.911089 28517 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:39:21.912197 28517 webserver.cc:460] Webserver started at http://127.25.71.66:41267/ using document root <none> and password file <none>
I20260928 16:39:21.912842 28517 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:21.912921 28517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:21.913132 28517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:21.915589 28517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "44ff3d85aacb46428c5c13b80c4dff78"
format_stamp: "Formatted at 2026-09-28 16:39:21 on dist-test-slave-nqgh"
server_key: "8bdc721255aa3f975bf9e4b945f12701"
server_key_iv: "cbb725939eec55664effa156b529f093"
server_key_version: "encryptionkey@0"
I20260928 16:39:21.916180 28517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "44ff3d85aacb46428c5c13b80c4dff78"
format_stamp: "Formatted at 2026-09-28 16:39:21 on dist-test-slave-nqgh"
server_key: "8bdc721255aa3f975bf9e4b945f12701"
server_key_iv: "cbb725939eec55664effa156b529f093"
server_key_version: "encryptionkey@0"
I20260928 16:39:21.920717 28517 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:21.924126 28532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:21.925882 28517 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:21.926057 28517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "44ff3d85aacb46428c5c13b80c4dff78"
format_stamp: "Formatted at 2026-09-28 16:39:21 on dist-test-slave-nqgh"
server_key: "8bdc721255aa3f975bf9e4b945f12701"
server_key_iv: "cbb725939eec55664effa156b529f093"
server_key_version: "encryptionkey@0"
I20260928 16:39:21.926193 28517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:21.961956 28517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:21.966311 28517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:21.966662 28517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:21.967453 28517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:21.968474 28517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:21.968531 28517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:21.968592 28517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:21.968621 28517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:21.979354 28517 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:41127
I20260928 16:39:21.979499 28645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:41127 every 8 connection(s)
I20260928 16:39:21.980446 28517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:39:21.986163 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 28517
I20260928 16:39:21.986294 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:39:21.986581 25885 external_mini_cluster.cc:1428] Setting key a1f658387f8015bd71d3ce936fdb0d2b
Sep 28 16:39:21 dist-test-slave-nqgh krb5kdc[28276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613561, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.71.67@KRBTEST.COM; defaulting to no policy
I20260928 16:39:21.998728 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:21.983106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:35325 (local address 127.25.71.126:46403)
0928 16:39:21.983294 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:21.983297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:21.984469 (+ 1172us) server_negotiation.cc:407] Connection header received
0928 16:39:21.986157 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:21.986161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:21.986239 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:21.986321 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:21.988060 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:21.988677 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:21.990606 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:21.990997 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:21.991226 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:21.994871 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:21.994875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:21.994877 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:21.994910 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:21.996322 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:21.996982 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:21.996985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:21.996986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:21.997072 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:21.997427 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:21.997431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:21.997433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:21.997641 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:21.997732 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:21.998332 (+ 600us) server_negotiation.cc:299] Negotiation successful
0928 16:39:21.998500 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 16:39:21.999658 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:21.983506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:46403 (local address 127.25.71.66:35325)
0928 16:39:21.984231 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:21.984303 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:21.985863 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:21.986582 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:21.986591 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:21.987135 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:21.987872 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:21.987889 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:21.988857 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:21.988861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:21.990397 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:21.990412 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:21.991291 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:21.991295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:21.991499 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:21.992331 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:21.992354 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:21.994585 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:21.996484 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:21.996492 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:21.996513 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:21.996866 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:21.997160 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:21.997163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:21.997165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:21.997327 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:21.997749 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:21.997758 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:21.998064 (+ 306us) client_negotiation.cc:769] Sending connection context
0928 16:39:21.998275 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 16:39:21.998710 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":162,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.67@KRBTEST.COM" created.
I20260928 16:39:22.001451 28646 heartbeater.cc:347] Connected to a master server at 127.25.71.126:46403
I20260928 16:39:22.001854 28646 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:22.002588 28646 heartbeater.cc:511] Master 127.25.71.126:46403 requested a full tablet report, sending...
I20260928 16:39:22.004451 28321 ts_manager.cc:194] Registered new tserver with Master: 44ff3d85aacb46428c5c13b80c4dff78 (127.25.71.66:41127)
I20260928 16:39:22.005390 28321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:35325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
I20260928 16:39:22.024053 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:46403
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:39659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:22.175699 28651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:22.176096 28651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:22.176149 28651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:22.180848 28651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:22.180968 28651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:22.181039 28651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:39:22.187714 28651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:39659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:22.189301 28651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:22.190755 28651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:39:22.205201 28651 server_base.cc:1034] running on GCE node
W20260928 16:39:22.204952 28656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:22.204967 28659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:22.204974 28657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:22.206221 28651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:22.207134 28651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:22.208550 28651 hybrid_clock.cc:648] HybridClock initialized: now 1790613562208508 us; error 70 us; skew 500 ppm
Sep 28 16:39:22 dist-test-slave-nqgh krb5kdc[28276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613562, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:22.212774 28651 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:39:22.214196 28651 webserver.cc:460] Webserver started at http://127.25.71.67:40141/ using document root <none> and password file <none>
I20260928 16:39:22.215070 28651 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:22.215173 28651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:22.215420 28651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:22.217835 28651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "3762bb28569a4f63912ca9211c2cb264"
format_stamp: "Formatted at 2026-09-28 16:39:22 on dist-test-slave-nqgh"
server_key: "b7a02018b3f5b5b69caae97def9deb2e"
server_key_iv: "2c67b9930f28877c1124d3a7bfda22a2"
server_key_version: "encryptionkey@0"
I20260928 16:39:22.218704 28651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "3762bb28569a4f63912ca9211c2cb264"
format_stamp: "Formatted at 2026-09-28 16:39:22 on dist-test-slave-nqgh"
server_key: "b7a02018b3f5b5b69caae97def9deb2e"
server_key_iv: "2c67b9930f28877c1124d3a7bfda22a2"
server_key_version: "encryptionkey@0"
I20260928 16:39:22.223958 28651 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 16:39:22.227706 28666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:22.229712 28651 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:22.229929 28651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "3762bb28569a4f63912ca9211c2cb264"
format_stamp: "Formatted at 2026-09-28 16:39:22 on dist-test-slave-nqgh"
server_key: "b7a02018b3f5b5b69caae97def9deb2e"
server_key_iv: "2c67b9930f28877c1124d3a7bfda22a2"
server_key_version: "encryptionkey@0"
I20260928 16:39:22.230252 28651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:22.246771 28651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:22.251052 28651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:22.251267 28651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:22.251945 28651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:22.252943 28651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:22.253000 28651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:22.253058 28651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:22.253086 28651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:22.265947 28651 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:41641
I20260928 16:39:22.265983 28779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:41641 every 8 connection(s)
I20260928 16:39:22.267148 28651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:39:22.274237 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 28651
I20260928 16:39:22.274395 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:39:22.274784 25885 external_mini_cluster.cc:1428] Setting key 9d8a0a3299df9f9cb680c357c5b7c104
Sep 28 16:39:22 dist-test-slave-nqgh krb5kdc[28276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613562, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:22.284421 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:22.269844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:55387 (local address 127.25.71.126:46403)
0928 16:39:22.270067 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:22.270072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:22.271327 (+ 1255us) server_negotiation.cc:407] Connection header received
0928 16:39:22.272835 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:22.272839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:22.272911 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:22.272985 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:22.274406 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:22.275087 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:22.276415 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:22.276647 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:22.276754 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:22.279970 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:22.279973 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:22.279975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:22.280007 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:22.281905 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:22.282570 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:22.282573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:22.282574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:22.282688 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:22.283141 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:22.283145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:22.283147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:22.283365 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:22.283458 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:22.284093 (+ 635us) server_negotiation.cc:299] Negotiation successful
0928 16:39:22.284245 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 16:39:22.285542 28782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:22.270283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:46403 (local address 127.25.71.67:55387)
0928 16:39:22.271078 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:22.271145 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:22.272568 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:22.273212 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:22.273221 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:22.273650 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:22.274205 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:22.274221 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:22.275234 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:22.275245 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:22.276221 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:22.276236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:22.276821 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:22.276825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:22.277013 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:22.277831 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:22.277855 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:22.279672 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:22.282088 (+ 2416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:22.282096 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:22.282115 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:22.282447 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:22.282785 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:22.282788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:22.282790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:22.283022 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:22.283487 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:22.283501 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:22.283794 (+ 293us) client_negotiation.cc:769] Sending connection context
0928 16:39:22.284027 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 16:39:22.284427 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":235,"threads_started":1}
I20260928 16:39:22.287488 28780 heartbeater.cc:347] Connected to a master server at 127.25.71.126:46403
I20260928 16:39:22.287914 28780 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:22.288825 28780 heartbeater.cc:511] Master 127.25.71.126:46403 requested a full tablet report, sending...
I20260928 16:39:22.290483 28321 ts_manager.cc:194] Registered new tserver with Master: 3762bb28569a4f63912ca9211c2cb264 (127.25.71.67:41641)
I20260928 16:39:22.291162 28321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:55387
I20260928 16:39:22.303407 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 16:39:22.321601 28360 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:22.319773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40990 (local address 127.25.71.126:46403)
0928 16:39:22.320000 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:22.320006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:22.320256 (+ 250us) server_negotiation.cc:407] Connection header received
0928 16:39:22.320414 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:22.320417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:22.320465 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:22.320536 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:22.321460 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40990: BlockingRecv error: recv got EOF from 127.0.0.1:40990 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 16:39:22.322863 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 28383
I20260928 16:39:22.329367 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 28517
I20260928 16:39:22.335685 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 28651
I20260928 16:39:22.341995 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 28289
2026-09-28T16:39:22Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2334 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:39:22 dist-test-slave-nqgh krb5kdc[28792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:39:22 dist-test-slave-nqgh krb5kdc[28792](info): set up 2 sockets
Sep 28 16:39:22 dist-test-slave-nqgh krb5kdc[28792](info): commencing operation
krb5kdc: starting...
W20260928 16:39:23.324905 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:39:23 dist-test-slave-nqgh krb5kdc[28792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:39:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:39:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:39:23.468734 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:43367
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:43367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:23.601393 28805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:23.601727 28805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:23.601770 28805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:23.606423 28805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:23.606531 28805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:23.606554 28805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:23.606572 28805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:23.606588 28805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:39:23.613072 28805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:43367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:43367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:23.614322 28805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:23.615473 28805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:23.626116 28810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:23.626116 28813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:23.626271 28805 server_base.cc:1034] running on GCE node
W20260928 16:39:23.626578 28811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:23.627308 28805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:23.628664 28805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:23.629963 28805 hybrid_clock.cc:648] HybridClock initialized: now 1790613563629942 us; error 60 us; skew 500 ppm
Sep 28 16:39:23 dist-test-slave-nqgh krb5kdc[28792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613563, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:23.641232 28805 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:39:23.643245 28805 webserver.cc:460] Webserver started at http://127.25.71.126:37807/ using document root <none> and password file <none>
I20260928 16:39:23.644197 28805 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:23.644325 28805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:23.644604 28805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:23.647166 28805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "02691e878fc944a9a272856327964953"
format_stamp: "Formatted at 2026-09-28 16:39:23 on dist-test-slave-nqgh"
server_key: "e947f673a307d060cc0b6e7acfec164f"
server_key_iv: "b8cb84cf827a2b62796ad16f1a61e7a1"
server_key_version: "encryptionkey@0"
I20260928 16:39:23.647843 28805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "02691e878fc944a9a272856327964953"
format_stamp: "Formatted at 2026-09-28 16:39:23 on dist-test-slave-nqgh"
server_key: "e947f673a307d060cc0b6e7acfec164f"
server_key_iv: "b8cb84cf827a2b62796ad16f1a61e7a1"
server_key_version: "encryptionkey@0"
I20260928 16:39:23.654115 28805 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 16:39:23.657939 28820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:23.660087 28805 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:23.660305 28805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "02691e878fc944a9a272856327964953"
format_stamp: "Formatted at 2026-09-28 16:39:23 on dist-test-slave-nqgh"
server_key: "e947f673a307d060cc0b6e7acfec164f"
server_key_iv: "b8cb84cf827a2b62796ad16f1a61e7a1"
server_key_version: "encryptionkey@0"
I20260928 16:39:23.660497 28805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:23.693351 28805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:23.697252 28805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:23.697444 28805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:23.706137 28805 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:43367
I20260928 16:39:23.706619 28872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:43367 every 8 connection(s)
I20260928 16:39:23.707544 28805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:39:23.709669 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 28805
I20260928 16:39:23.709807 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:39:23.710110 25885 external_mini_cluster.cc:1428] Setting key c36ddc59892dfa4ae6214450e5c63c65
I20260928 16:39:23.712332 28873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:23.721215 28873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953: Bootstrap starting.
Sep 28 16:39:23 dist-test-slave-nqgh krb5kdc[28792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:23.724506 28873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:23.725462 28873 log.cc:824] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:23.728048 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:23.712117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58406 (local address 127.25.71.126:43367)
0928 16:39:23.712675 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:23.712693 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:23.712735 (+ 42us) server_negotiation.cc:407] Connection header received
0928 16:39:23.713471 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:23.713492 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:23.713916 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:23.714263 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:23.715564 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:23.716751 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:23.718214 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:23.718963 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:23.719152 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:23.722751 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:23.722758 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:23.722778 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:23.722825 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:23.724850 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:23.725372 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:23.725378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:23.725385 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:23.725524 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:23.725788 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:23.725792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:23.725793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:23.726283 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:23.726499 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:23.726831 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 16:39:23.727185 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":176,"threads_started":1}
I20260928 16:39:23.728279 28873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953: No bootstrap required, opened a new log
I20260928 16:39:23.731389 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:39:23.732826 28873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02691e878fc944a9a272856327964953" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 43367 } }
I20260928 16:39:23.733179 28873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:23.733253 28873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02691e878fc944a9a272856327964953, State: Initialized, Role: FOLLOWER
I20260928 16:39:23.733817 28873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02691e878fc944a9a272856327964953" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 43367 } }
I20260928 16:39:23.733984 28873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:39:23.734063 28873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:39:23.734189 28873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:23.735805 28873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02691e878fc944a9a272856327964953" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 43367 } }
I20260928 16:39:23.736315 28873 leader_election.cc:302] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02691e878fc944a9a272856327964953; no voters:
I20260928 16:39:23.736815 28873 leader_election.cc:288] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:39:23.737047 28879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:23.737398 28879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953 [term 1 LEADER]: Becoming Leader. State: Replica: 02691e878fc944a9a272856327964953, State: Running, Role: LEADER
I20260928 16:39:23.737926 28879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02691e878fc944a9a272856327964953" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 43367 } }
I20260928 16:39:23.738518 28873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:23.740060 28880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02691e878fc944a9a272856327964953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02691e878fc944a9a272856327964953" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 43367 } } }
I20260928 16:39:23.740248 28880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:23.740577 28881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02691e878fc944a9a272856327964953. Latest consensus state: current_term: 1 leader_uuid: "02691e878fc944a9a272856327964953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02691e878fc944a9a272856327964953" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 43367 } } }
I20260928 16:39:23.740700 28881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:23.741007 28890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:39:23.745083 28890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:39:23.752178 28890 catalog_manager.cc:1347] Generated new cluster ID: 7c8474aa47b74424b534a9112f817375
I20260928 16:39:23.752281 28890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:39:23.756712 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:23.749891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.25.71.126:43367)
0928 16:39:23.750046 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:23.750052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:23.750435 (+ 383us) server_negotiation.cc:407] Connection header received
0928 16:39:23.750561 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:23.750564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:23.750612 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:23.750729 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:23.751623 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:23.752424 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:23.753852 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:23.754240 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:23.754361 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:23.755559 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:23.755563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:23.755566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:39:23.756168 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:23.756212 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:39:23.756247 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:23.756344 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:23.756474 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 16:39:23.756554 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:39:23.768599 28890 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:39:23.769953 28890 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:39:23.779843 28890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02691e878fc944a9a272856327964953: Generated new TSK 0
I20260928 16:39:23.780691 28890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:39:23.791174 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 28805
2026-09-28T16:39:23Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1447 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:39:23 dist-test-slave-nqgh krb5kdc[28906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:39:23 dist-test-slave-nqgh krb5kdc[28906](info): set up 2 sockets
Sep 28 16:39:23 dist-test-slave-nqgh krb5kdc[28906](info): commencing operation
krb5kdc: starting...
W20260928 16:39:24.776538 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:39:24 dist-test-slave-nqgh krb5kdc[28906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:39:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:39:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:39:24.916056 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:43139
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:43139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:25.051489 28919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:25.051923 28919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:25.051993 28919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:25.056739 28919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:25.056844 28919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:25.056874 28919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:25.056896 28919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:25.056916 28919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:39:25.063638 28919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:43139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:43139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:25.065245 28919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:25.066608 28919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:39:25.077925 28919 server_base.cc:1034] running on GCE node
W20260928 16:39:25.078074 28924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:25.077829 28925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:25.077838 28927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:25.079216 28919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:25.080852 28919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:25.082218 28919 hybrid_clock.cc:648] HybridClock initialized: now 1790613565082179 us; error 66 us; skew 500 ppm
Sep 28 16:39:25 dist-test-slave-nqgh krb5kdc[28906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613565, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:25.086282 28919 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:39:25.087693 28919 webserver.cc:460] Webserver started at http://127.25.71.126:38377/ using document root <none> and password file <none>
I20260928 16:39:25.088455 28919 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:25.088549 28919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:25.088824 28919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:25.091624 28919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "233e966027874aeca4916f5eaff424be"
format_stamp: "Formatted at 2026-09-28 16:39:25 on dist-test-slave-nqgh"
server_key: "a8971d5589c120dee5ed204409f89329"
server_key_iv: "f29a17ddb35a7c9883c13f14191a8306"
server_key_version: "encryptionkey@0"
I20260928 16:39:25.092372 28919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "233e966027874aeca4916f5eaff424be"
format_stamp: "Formatted at 2026-09-28 16:39:25 on dist-test-slave-nqgh"
server_key: "a8971d5589c120dee5ed204409f89329"
server_key_iv: "f29a17ddb35a7c9883c13f14191a8306"
server_key_version: "encryptionkey@0"
I20260928 16:39:25.097810 28919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:39:25.101768 28934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:25.103901 28919 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:25.104116 28919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "233e966027874aeca4916f5eaff424be"
format_stamp: "Formatted at 2026-09-28 16:39:25 on dist-test-slave-nqgh"
server_key: "a8971d5589c120dee5ed204409f89329"
server_key_iv: "f29a17ddb35a7c9883c13f14191a8306"
server_key_version: "encryptionkey@0"
I20260928 16:39:25.104307 28919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:25.123083 28919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:25.127547 28919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:25.127799 28919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:25.137434 28919 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:43139
I20260928 16:39:25.137853 28986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:43139 every 8 connection(s)
I20260928 16:39:25.138908 28919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:39:25.142580 28987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:25.145233 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 28919
I20260928 16:39:25.145404 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:39:25.145696 25885 external_mini_cluster.cc:1428] Setting key 82bd377fa3eb0af4cfc70a6e23d2b903
I20260928 16:39:25.150324 28987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be: Bootstrap starting.
I20260928 16:39:25.152878 28987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:25.153927 28987 log.cc:824] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:25.156543 28987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be: No bootstrap required, opened a new log
Sep 28 16:39:25 dist-test-slave-nqgh krb5kdc[28906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:25.160225 28987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233e966027874aeca4916f5eaff424be" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 43139 } }
I20260928 16:39:25.160537 28987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:25.160590 28987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 233e966027874aeca4916f5eaff424be, State: Initialized, Role: FOLLOWER
I20260928 16:39:25.161077 28987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233e966027874aeca4916f5eaff424be" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 43139 } }
I20260928 16:39:25.161219 28987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:39:25.161290 28987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:39:25.161397 28987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:25.162477 28987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233e966027874aeca4916f5eaff424be" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 43139 } }
I20260928 16:39:25.162909 28987 leader_election.cc:302] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 233e966027874aeca4916f5eaff424be; no voters:
I20260928 16:39:25.163255 28987 leader_election.cc:288] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:39:25.163367 28992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:25.163579 28992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be [term 1 LEADER]: Becoming Leader. State: Replica: 233e966027874aeca4916f5eaff424be, State: Running, Role: LEADER
I20260928 16:39:25.164018 28992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233e966027874aeca4916f5eaff424be" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 43139 } }
I20260928 16:39:25.164954 28987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:25.165333 28990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:25.147710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45934 (local address 127.25.71.126:43139)
0928 16:39:25.148410 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:25.148444 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:25.148504 (+ 60us) server_negotiation.cc:407] Connection header received
0928 16:39:25.149530 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:25.149559 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:25.150072 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:25.150544 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:25.151607 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:25.152794 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:25.153817 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:25.154433 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:25.154615 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:25.157892 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:25.157901 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:25.157924 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:25.157970 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:25.161040 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:25.161952 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:25.161959 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:25.161966 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:25.162127 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:25.162410 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:25.162413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:25.162414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:25.163049 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:25.163299 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:25.163776 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 16:39:25.164256 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":172,"threads_started":1}
I20260928 16:39:25.167426 28993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 233e966027874aeca4916f5eaff424be. Latest consensus state: current_term: 1 leader_uuid: "233e966027874aeca4916f5eaff424be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233e966027874aeca4916f5eaff424be" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 43139 } } }
I20260928 16:39:25.167793 28993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:25.167898 28994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "233e966027874aeca4916f5eaff424be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233e966027874aeca4916f5eaff424be" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 43139 } } }
I20260928 16:39:25.167981 28994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be [sys.catalog]: This master's current role is: LEADER
W20260928 16:39:25.171064 29006 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:39:25.171200 29006 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:39:25.171276 29007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:39:25.172320 29007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:39:25.180184 29007 catalog_manager.cc:1347] Generated new cluster ID: 06865ba875624d238534928f94dc7f21
I20260928 16:39:25.180294 29007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:39:25.188484 29007 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:39:25.190023 29007 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:39:25.199596 29007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 233e966027874aeca4916f5eaff424be: Generated new TSK 0
I20260928 16:39:25.200407 29007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
I20260928 16:39:25.247042 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:43139
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:25.376919 29013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:25.377256 29013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:25.377321 29013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:25.381970 29013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:25.382063 29013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:25.382161 29013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:39:25.388190 29013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:43139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:25.389446 29013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:25.390571 29013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:25.402024 29021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:25.402029 29018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:25.402190 29013 server_base.cc:1034] running on GCE node
W20260928 16:39:25.402029 29019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:25.403005 29013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:25.403764 29013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:25.404984 29013 hybrid_clock.cc:648] HybridClock initialized: now 1790613565404989 us; error 95 us; skew 500 ppm
Sep 28 16:39:25 dist-test-slave-nqgh krb5kdc[28906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613565, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:25.408628 29013 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:39:25.409905 29013 webserver.cc:460] Webserver started at http://127.25.71.65:45563/ using document root <none> and password file <none>
I20260928 16:39:25.410578 29013 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:25.410694 29013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:25.410912 29013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:25.413237 29013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "020143d9457d4af9b76ad2305675adde"
format_stamp: "Formatted at 2026-09-28 16:39:25 on dist-test-slave-nqgh"
server_key: "07f4758cb959defb2af0aac2a865d293"
server_key_iv: "edb9e97a63d050392657b36200d82426"
server_key_version: "encryptionkey@0"
I20260928 16:39:25.413838 29013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "020143d9457d4af9b76ad2305675adde"
format_stamp: "Formatted at 2026-09-28 16:39:25 on dist-test-slave-nqgh"
server_key: "07f4758cb959defb2af0aac2a865d293"
server_key_iv: "edb9e97a63d050392657b36200d82426"
server_key_version: "encryptionkey@0"
I20260928 16:39:25.418500 29013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260928 16:39:25.421808 29028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:25.423766 29013 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:25.423951 29013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "020143d9457d4af9b76ad2305675adde"
format_stamp: "Formatted at 2026-09-28 16:39:25 on dist-test-slave-nqgh"
server_key: "07f4758cb959defb2af0aac2a865d293"
server_key_iv: "edb9e97a63d050392657b36200d82426"
server_key_version: "encryptionkey@0"
I20260928 16:39:25.424077 29013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:25.436350 29013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:25.440510 29013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:25.440694 29013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:25.441334 29013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:25.442303 29013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:25.442361 29013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:25.442406 29013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:25.442421 29013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:25.453612 29013 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:44597
I20260928 16:39:25.453668 29141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:44597 every 8 connection(s)
I20260928 16:39:25.454782 29013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:39:25.455744 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 29013
I20260928 16:39:25.455873 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:39:25.456192 25885 external_mini_cluster.cc:1428] Setting key 2dde5fa69373f4d100da80e8824ff8b9
WARNING: no policy specified for kudu/127.25.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.66@KRBTEST.COM" created.
Sep 28 16:39:25 dist-test-slave-nqgh krb5kdc[28906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613565, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:25.476600 28990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:25.457361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:50629 (local address 127.25.71.126:43139)
0928 16:39:25.457551 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:25.457558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:25.459759 (+ 2201us) server_negotiation.cc:407] Connection header received
0928 16:39:25.461216 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:25.461220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:25.461294 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:25.461412 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:25.463278 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:25.463956 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:25.465958 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:25.466286 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:25.466422 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:25.472267 (+ 5845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:25.472272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:25.472275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:25.472316 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:25.473862 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:25.474599 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:25.474603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:25.474604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:25.474745 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:25.475155 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:25.475158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:25.475160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:25.475402 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:25.475499 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:25.476177 (+ 678us) server_negotiation.cc:299] Negotiation successful
0928 16:39:25.476378 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:39:25.477715 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:25.457885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:43139 (local address 127.25.71.65:50629)
0928 16:39:25.459541 (+ 1656us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:25.459606 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:25.460940 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:25.461695 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:25.461705 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:25.462215 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:25.463030 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:25.463054 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:25.464154 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:25.464159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:25.465748 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:25.465765 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:25.466555 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:25.466560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:25.466839 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:25.467625 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:25.467644 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:25.471930 (+ 4286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:25.474048 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:25.474058 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:25.474080 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:25.474473 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:25.474842 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:25.474846 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:25.474848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:25.475045 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:25.475513 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:25.475523 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:25.475885 (+ 362us) client_negotiation.cc:769] Sending connection context
0928 16:39:25.476100 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 16:39:25.476593 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1372,"thread_start_us":154,"threads_started":1}
I20260928 16:39:25.479625 29142 heartbeater.cc:347] Connected to a master server at 127.25.71.126:43139
I20260928 16:39:25.480006 29142 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:25.480675 29142 heartbeater.cc:511] Master 127.25.71.126:43139 requested a full tablet report, sending...
I20260928 16:39:25.482758 28951 ts_manager.cc:194] Registered new tserver with Master: 020143d9457d4af9b76ad2305675adde (127.25.71.65:44597)
I20260928 16:39:25.484338 28951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:50629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
I20260928 16:39:25.487759 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:43139
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:25.620232 29147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:25.620640 29147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:25.620694 29147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:25.625331 29147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:25.625432 29147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:25.625504 29147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:39:25.632088 29147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:43139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:25.633589 29147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:25.634912 29147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:25.647665 29152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:25.647655 29153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:25.647655 29155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:25.647976 29147 server_base.cc:1034] running on GCE node
I20260928 16:39:25.648774 29147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:25.649680 29147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:25.650909 29147 hybrid_clock.cc:648] HybridClock initialized: now 1790613565650884 us; error 45 us; skew 500 ppm
Sep 28 16:39:25 dist-test-slave-nqgh krb5kdc[28906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613565, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:25.654781 29147 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:39:25.656203 29147 webserver.cc:460] Webserver started at http://127.25.71.66:38467/ using document root <none> and password file <none>
I20260928 16:39:25.656967 29147 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:25.657055 29147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:25.657306 29147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:25.660080 29147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "2f46a4f7390a41dc89985d6b7cfddfe1"
format_stamp: "Formatted at 2026-09-28 16:39:25 on dist-test-slave-nqgh"
server_key: "b07c0e2b7b916c4d7bde1e054235ae3d"
server_key_iv: "436af0b4e2a9b13706d21adbb31cb0bb"
server_key_version: "encryptionkey@0"
I20260928 16:39:25.660799 29147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "2f46a4f7390a41dc89985d6b7cfddfe1"
format_stamp: "Formatted at 2026-09-28 16:39:25 on dist-test-slave-nqgh"
server_key: "b07c0e2b7b916c4d7bde1e054235ae3d"
server_key_iv: "436af0b4e2a9b13706d21adbb31cb0bb"
server_key_version: "encryptionkey@0"
I20260928 16:39:25.665746 29147 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:39:25.669265 29162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:25.671275 29147 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:25.671476 29147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "2f46a4f7390a41dc89985d6b7cfddfe1"
format_stamp: "Formatted at 2026-09-28 16:39:25 on dist-test-slave-nqgh"
server_key: "b07c0e2b7b916c4d7bde1e054235ae3d"
server_key_iv: "436af0b4e2a9b13706d21adbb31cb0bb"
server_key_version: "encryptionkey@0"
I20260928 16:39:25.671636 29147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:25.689594 29147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:25.693372 29147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:25.693608 29147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:25.694391 29147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:25.695505 29147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:25.695559 29147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:25.695623 29147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:25.695648 29147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:25.707827 29147 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:45773
I20260928 16:39:25.707940 29275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:45773 every 8 connection(s)
I20260928 16:39:25.709151 29147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:39:25.716564 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 29147
I20260928 16:39:25.716693 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:39:25.717026 25885 external_mini_cluster.cc:1428] Setting key 9a56240151bb466751f4342f681f8417
Sep 28 16:39:25 dist-test-slave-nqgh krb5kdc[28906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613565, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.71.67@KRBTEST.COM; defaulting to no policy
I20260928 16:39:25.729180 28990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:25.711660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:44003 (local address 127.25.71.126:43139)
0928 16:39:25.711983 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:25.711988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:25.713284 (+ 1296us) server_negotiation.cc:407] Connection header received
0928 16:39:25.715406 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:25.715410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:25.715495 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:25.715607 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:25.718177 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:25.718936 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:25.720406 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:25.720718 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:25.720819 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:25.724548 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:25.724554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:25.724557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:25.724590 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:25.726490 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:25.727184 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:25.727188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:25.727191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:25.727276 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:25.727637 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:25.727640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:25.727642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:25.727865 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:25.727955 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:25.728686 (+ 731us) server_negotiation.cc:299] Negotiation successful
0928 16:39:25.728939 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":174}
I20260928 16:39:25.730054 29278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:25.712312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:43139 (local address 127.25.71.66:44003)
0928 16:39:25.713036 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:25.713107 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:25.714984 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:25.715905 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:25.715918 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:25.716581 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:25.717535 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:25.717562 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:25.719080 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:25.719083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:25.720217 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:25.720230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:25.720929 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:25.720933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:25.721204 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:25.721986 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:25.722011 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:25.724302 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:25.726703 (+ 2401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:25.726712 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:25.726734 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:25.727075 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:25.727373 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:25.727376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:25.727378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:25.727535 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:25.727969 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:25.727977 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:25.728418 (+ 441us) client_negotiation.cc:769] Sending connection context
0928 16:39:25.728606 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 16:39:25.729006 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":162,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.67@KRBTEST.COM" created.
I20260928 16:39:25.731877 29276 heartbeater.cc:347] Connected to a master server at 127.25.71.126:43139
I20260928 16:39:25.732143 29276 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:25.732681 29276 heartbeater.cc:511] Master 127.25.71.126:43139 requested a full tablet report, sending...
I20260928 16:39:25.733973 28951 ts_manager.cc:194] Registered new tserver with Master: 2f46a4f7390a41dc89985d6b7cfddfe1 (127.25.71.66:45773)
I20260928 16:39:25.734972 28951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:44003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
I20260928 16:39:25.751173 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:43139
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:25.890800 29281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:25.891211 29281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:25.891263 29281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:25.895776 29281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:25.895874 29281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:25.895944 29281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:39:25.902210 29281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:43139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:25.903874 29281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:25.905227 29281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:25.917881 29286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:25.917852 29289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:25.917955 29281 server_base.cc:1034] running on GCE node
W20260928 16:39:25.917881 29287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:25.918928 29281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:25.919859 29281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:25.921187 29281 hybrid_clock.cc:648] HybridClock initialized: now 1790613565921153 us; error 51 us; skew 500 ppm
Sep 28 16:39:25 dist-test-slave-nqgh krb5kdc[28906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613565, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:25.924858 29281 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:39:25.925990 29281 webserver.cc:460] Webserver started at http://127.25.71.67:38357/ using document root <none> and password file <none>
I20260928 16:39:25.926682 29281 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:25.926762 29281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:25.927045 29281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:25.929543 29281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "f20cfd886b614c58bb905f1b694ba1f8"
format_stamp: "Formatted at 2026-09-28 16:39:25 on dist-test-slave-nqgh"
server_key: "1cf893d6e1884dd0c53770e600f77ca0"
server_key_iv: "9c149813ca694fd78c2cb62ee4bcf8ea"
server_key_version: "encryptionkey@0"
I20260928 16:39:25.930122 29281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "f20cfd886b614c58bb905f1b694ba1f8"
format_stamp: "Formatted at 2026-09-28 16:39:25 on dist-test-slave-nqgh"
server_key: "1cf893d6e1884dd0c53770e600f77ca0"
server_key_iv: "9c149813ca694fd78c2cb62ee4bcf8ea"
server_key_version: "encryptionkey@0"
I20260928 16:39:25.935012 29281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 16:39:25.938326 29296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:25.940414 29281 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:25.940589 29281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "f20cfd886b614c58bb905f1b694ba1f8"
format_stamp: "Formatted at 2026-09-28 16:39:25 on dist-test-slave-nqgh"
server_key: "1cf893d6e1884dd0c53770e600f77ca0"
server_key_iv: "9c149813ca694fd78c2cb62ee4bcf8ea"
server_key_version: "encryptionkey@0"
I20260928 16:39:25.940737 29281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:25.970853 29281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:25.975379 29281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:25.975612 29281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:25.976521 29281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:25.977691 29281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:25.977751 29281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:25.977802 29281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:25.977816 29281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:25.989941 29281 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:38459
I20260928 16:39:25.990011 29410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:38459 every 8 connection(s)
I20260928 16:39:25.991226 29281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:39:26.000609 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 29281
I20260928 16:39:26.000748 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:39:26.001044 25885 external_mini_cluster.cc:1428] Setting key 36d2b9fccba267faef1d5acc2add568a
Sep 28 16:39:25 dist-test-slave-nqgh krb5kdc[28906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613565, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:26.008842 28990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:25.993702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:58959 (local address 127.25.71.126:43139)
0928 16:39:25.993893 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:25.993898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:25.995194 (+ 1296us) server_negotiation.cc:407] Connection header received
0928 16:39:25.996718 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:25.996722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:25.996786 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:25.996865 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:25.998323 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:25.999063 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:26.000420 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:26.000707 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:26.000860 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:26.004601 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:26.004605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:26.004607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:26.004644 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:26.006160 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:26.006914 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:26.006918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:26.006919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:26.007009 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:26.007395 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:26.007398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:26.007399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:26.007670 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:26.007779 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:26.008467 (+ 688us) server_negotiation.cc:299] Negotiation successful
0928 16:39:26.008641 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 16:39:26.010001 29413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:25.994202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:43139 (local address 127.25.71.67:58959)
0928 16:39:25.995001 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:25.995065 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:25.996473 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:25.997078 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:25.997086 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:25.997501 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:25.998073 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:25.998088 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:25.999244 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:25.999248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:26.000229 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:26.000241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:26.001079 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:26.001085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:26.001367 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:26.002275 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:26.002303 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:26.004372 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:26.006332 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:26.006341 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:26.006364 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:26.006802 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:26.007100 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:26.007104 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:26.007106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:26.007291 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:26.007780 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:26.007790 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:26.008191 (+ 401us) client_negotiation.cc:769] Sending connection context
0928 16:39:26.008397 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 16:39:26.008857 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":164,"threads_started":1}
I20260928 16:39:26.011610 29411 heartbeater.cc:347] Connected to a master server at 127.25.71.126:43139
I20260928 16:39:26.011986 29411 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:26.012642 29411 heartbeater.cc:511] Master 127.25.71.126:43139 requested a full tablet report, sending...
I20260928 16:39:26.014120 28951 ts_manager.cc:194] Registered new tserver with Master: f20cfd886b614c58bb905f1b694ba1f8 (127.25.71.67:38459)
I20260928 16:39:26.014886 28951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:58959
I20260928 16:39:26.016721 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:39:26 dist-test-slave-nqgh krb5kdc[28906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613566, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 16:39:26 dist-test-slave-nqgh krb5kdc[28906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613566, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:26.048713 28990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:26.037254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45950 (local address 127.25.71.126:43139)
0928 16:39:26.037511 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:26.037516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:26.037670 (+ 154us) server_negotiation.cc:407] Connection header received
0928 16:39:26.037894 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:26.037897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:26.037948 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:26.038023 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:26.038911 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:26.039455 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:26.040501 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:26.040804 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:26.040959 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:26.044377 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:26.044391 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:26.044393 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:26.044449 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:26.046326 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:26.047007 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:26.047012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:26.047014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:26.047123 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:26.047468 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:26.047471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:26.047473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:26.047847 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:26.047981 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:26.048312 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 16:39:26.048459 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260928 16:39:26.049197 28951 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45950
I20260928 16:39:26.050357 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 29013
I20260928 16:39:26.056555 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 29147
I20260928 16:39:26.063745 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 29281
I20260928 16:39:26.070267 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 28919
2026-09-28T16:39:26Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2280 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:39:26 dist-test-slave-nqgh krb5kdc[29423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:39:26 dist-test-slave-nqgh krb5kdc[29423](info): set up 2 sockets
Sep 28 16:39:26 dist-test-slave-nqgh krb5kdc[29423](info): commencing operation
krb5kdc: starting...
W20260928 16:39:27.044519 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:39:27 dist-test-slave-nqgh krb5kdc[29423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:39:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:39:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:39:27.199050 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:42977
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:42977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:27.335654 29436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:27.336098 29436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:27.336179 29436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:27.340827 29436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:27.340934 29436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:27.340965 29436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:27.340991 29436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:27.341012 29436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:39:27.347822 29436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:42977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:42977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:27.349462 29436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:27.350939 29436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:27.362066 29441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:27.362080 29444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:27.362080 29442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:27.362497 29436 server_base.cc:1034] running on GCE node
I20260928 16:39:27.363523 29436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:27.365012 29436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:27.366266 29436 hybrid_clock.cc:648] HybridClock initialized: now 1790613567366236 us; error 65 us; skew 500 ppm
Sep 28 16:39:27 dist-test-slave-nqgh krb5kdc[29423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613567, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:27.369735 29436 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:39:27.370905 29436 webserver.cc:460] Webserver started at http://127.25.71.126:32881/ using document root <none> and password file <none>
I20260928 16:39:27.371551 29436 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:27.371623 29436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:27.371815 29436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:27.374086 29436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "c4e904d252674bdb810e824e5e16ffdd"
format_stamp: "Formatted at 2026-09-28 16:39:27 on dist-test-slave-nqgh"
server_key: "d7e1f56be7eb2554efc91f04a4870e3f"
server_key_iv: "fbab7fe079d91f3cf88ec127260ac1b4"
server_key_version: "encryptionkey@0"
I20260928 16:39:27.374717 29436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "c4e904d252674bdb810e824e5e16ffdd"
format_stamp: "Formatted at 2026-09-28 16:39:27 on dist-test-slave-nqgh"
server_key: "d7e1f56be7eb2554efc91f04a4870e3f"
server_key_iv: "fbab7fe079d91f3cf88ec127260ac1b4"
server_key_version: "encryptionkey@0"
I20260928 16:39:27.379388 29436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:27.382697 29451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:27.384807 29436 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:27.384995 29436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "c4e904d252674bdb810e824e5e16ffdd"
format_stamp: "Formatted at 2026-09-28 16:39:27 on dist-test-slave-nqgh"
server_key: "d7e1f56be7eb2554efc91f04a4870e3f"
server_key_iv: "fbab7fe079d91f3cf88ec127260ac1b4"
server_key_version: "encryptionkey@0"
I20260928 16:39:27.385174 29436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:27.408807 29436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:27.412653 29436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:27.412853 29436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:27.421088 29436 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:42977
I20260928 16:39:27.421180 29503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:42977 every 8 connection(s)
I20260928 16:39:27.422228 29436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:39:27.426352 29504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:27.428409 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 29436
I20260928 16:39:27.428546 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:39:27.428844 25885 external_mini_cluster.cc:1428] Setting key fdcbdf41cdc10f7ec5e3352e8ead2415
I20260928 16:39:27.434830 29504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd: Bootstrap starting.
I20260928 16:39:27.438369 29504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:27.439559 29504 log.cc:824] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:39:27 dist-test-slave-nqgh krb5kdc[29423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:27.442902 29504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd: No bootstrap required, opened a new log
I20260928 16:39:27.446323 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:27.430541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60452 (local address 127.25.71.126:42977)
0928 16:39:27.431075 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:27.431093 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:27.431138 (+ 45us) server_negotiation.cc:407] Connection header received
0928 16:39:27.431981 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:27.432003 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:27.432393 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:27.432737 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:27.433954 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:27.435239 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:27.436624 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:27.437207 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:27.437365 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:27.440823 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:27.440829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:27.440848 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:27.440882 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:27.442945 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:27.443497 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:27.443503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:27.443509 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:27.443741 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:27.444040 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:27.444043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:27.444045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:27.444557 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:27.444750 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:27.445089 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 16:39:27.445438 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":141,"threads_started":1}
I20260928 16:39:27.448285 29504 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e904d252674bdb810e824e5e16ffdd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42977 } }
I20260928 16:39:27.448719 29504 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:27.448784 29504 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4e904d252674bdb810e824e5e16ffdd, State: Initialized, Role: FOLLOWER
I20260928 16:39:27.449388 29504 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e904d252674bdb810e824e5e16ffdd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42977 } }
I20260928 16:39:27.449704 29504 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:39:27.449833 29504 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:39:27.449961 29504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:27.451673 29504 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e904d252674bdb810e824e5e16ffdd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42977 } }
I20260928 16:39:27.452288 29504 leader_election.cc:302] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4e904d252674bdb810e824e5e16ffdd; no voters:
I20260928 16:39:27.452773 29504 leader_election.cc:288] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:39:27.453037 29509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:27.453359 29509 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd [term 1 LEADER]: Becoming Leader. State: Replica: c4e904d252674bdb810e824e5e16ffdd, State: Running, Role: LEADER
I20260928 16:39:27.453967 29509 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e904d252674bdb810e824e5e16ffdd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42977 } }
I20260928 16:39:27.454464 29504 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:27.456794 29510 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4e904d252674bdb810e824e5e16ffdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e904d252674bdb810e824e5e16ffdd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42977 } } }
I20260928 16:39:27.457023 29510 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:27.456804 29511 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4e904d252674bdb810e824e5e16ffdd. Latest consensus state: current_term: 1 leader_uuid: "c4e904d252674bdb810e824e5e16ffdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e904d252674bdb810e824e5e16ffdd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42977 } } }
I20260928 16:39:27.457696 29511 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd [sys.catalog]: This master's current role is: LEADER
W20260928 16:39:27.460884 29523 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:39:27.460994 29523 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:39:27.461104 29524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:39:27.461961 29524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:39:27.470325 29524 catalog_manager.cc:1347] Generated new cluster ID: 0209d4298d2146ebb40ee9b0a1f72931
I20260928 16:39:27.470451 29524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:39:27.496915 29524 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:39:27.498373 29524 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:39:27.509418 29524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4e904d252674bdb810e824e5e16ffdd: Generated new TSK 0
I20260928 16:39:27.510452 29524 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
I20260928 16:39:27.554869 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:42977
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:27.681731 29530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:27.682049 29530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:27.682096 29530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:27.686363 29530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:27.686455 29530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:27.686515 29530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:39:27.692770 29530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:42977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:27.693986 29530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:27.695106 29530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:27.706262 29535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:27.706281 29538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:27.706269 29536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:27.706642 29530 server_base.cc:1034] running on GCE node
I20260928 16:39:27.707434 29530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:27.708184 29530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:27.709393 29530 hybrid_clock.cc:648] HybridClock initialized: now 1790613567709360 us; error 48 us; skew 500 ppm
Sep 28 16:39:27 dist-test-slave-nqgh krb5kdc[29423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613567, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:27.712913 29530 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:39:27.714021 29530 webserver.cc:460] Webserver started at http://127.25.71.65:44383/ using document root <none> and password file <none>
I20260928 16:39:27.714687 29530 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:27.714764 29530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:27.714985 29530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:27.717221 29530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "804de557739342f9b8f88c9a4e3572f1"
format_stamp: "Formatted at 2026-09-28 16:39:27 on dist-test-slave-nqgh"
server_key: "2415b34f07ecc70490597ea357c6eec6"
server_key_iv: "c92d260d6931a968fa70adb3c00a70ea"
server_key_version: "encryptionkey@0"
I20260928 16:39:27.717849 29530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "804de557739342f9b8f88c9a4e3572f1"
format_stamp: "Formatted at 2026-09-28 16:39:27 on dist-test-slave-nqgh"
server_key: "2415b34f07ecc70490597ea357c6eec6"
server_key_iv: "c92d260d6931a968fa70adb3c00a70ea"
server_key_version: "encryptionkey@0"
I20260928 16:39:27.722280 29530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:39:27.726250 29545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:27.728143 29530 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260928 16:39:27.728315 29530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "804de557739342f9b8f88c9a4e3572f1"
format_stamp: "Formatted at 2026-09-28 16:39:27 on dist-test-slave-nqgh"
server_key: "2415b34f07ecc70490597ea357c6eec6"
server_key_iv: "c92d260d6931a968fa70adb3c00a70ea"
server_key_version: "encryptionkey@0"
I20260928 16:39:27.728459 29530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:27.746124 29530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:27.749788 29530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:27.750007 29530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:27.750717 29530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:27.751652 29530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:27.751704 29530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:27.751760 29530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:27.751789 29530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:27.761982 29530 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:43753
I20260928 16:39:27.762071 29658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:43753 every 8 connection(s)
I20260928 16:39:27.763106 29530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:39:27.763834 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 29530
I20260928 16:39:27.763985 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:39:27.764365 25885 external_mini_cluster.cc:1428] Setting key 0e3f99652dc6ed2eba7354897decc4ec
WARNING: no policy specified for kudu/127.25.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.66@KRBTEST.COM" created.
Sep 28 16:39:27 dist-test-slave-nqgh krb5kdc[29423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613567, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:27.787922 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:27.769954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:44703 (local address 127.25.71.126:42977)
0928 16:39:27.770164 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:27.770170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:27.770190 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:39:27.770260 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:27.770263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:27.770328 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:27.770449 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:27.772928 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:27.773843 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:27.776029 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:27.776424 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:27.776566 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:27.782270 (+ 5704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:27.782274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:27.782277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:27.782313 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:27.784214 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:27.785357 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:27.785361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:27.785363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:27.785470 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:27.786049 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:27.786053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:27.786054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:27.786360 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:27.786454 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:27.787515 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0928 16:39:27.787707 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 16:39:27.789155 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:27.766258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:42977 (local address 127.25.71.65:44703)
0928 16:39:27.766924 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:27.766984 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:27.768685 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:27.770968 (+ 2283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:27.770981 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:27.771668 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:27.772634 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:27.772665 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:27.774084 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:27.774088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:27.775811 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:27.775834 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:27.776870 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:27.776876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:27.777200 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:27.778031 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:27.778057 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:27.781969 (+ 3912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:27.784502 (+ 2533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:27.784515 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:27.784549 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:27.785143 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:27.785576 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:27.785579 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:27.785582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:27.785900 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:27.786502 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:27.786513 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:27.787116 (+ 603us) client_negotiation.cc:769] Sending connection context
0928 16:39:27.787411 (+ 295us) client_negotiation.cc:240] Negotiation successful
0928 16:39:27.787871 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":207,"threads_started":1}
I20260928 16:39:27.791021 29659 heartbeater.cc:347] Connected to a master server at 127.25.71.126:42977
I20260928 16:39:27.791366 29659 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:27.792066 29659 heartbeater.cc:511] Master 127.25.71.126:42977 requested a full tablet report, sending...
I20260928 16:39:27.794248 29468 ts_manager.cc:194] Registered new tserver with Master: 804de557739342f9b8f88c9a4e3572f1 (127.25.71.65:43753)
I20260928 16:39:27.795830 29468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:44703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
I20260928 16:39:27.803627 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:42977
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:27.939808 29664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:27.940433 29664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:27.940536 29664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:27.948994 29664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:27.949173 29664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:27.949245 29664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:39:27.960844 29664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:42977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:27.963394 29664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:27.966427 29664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:27.986675 29669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:27.986794 29670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:27.987190 29672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:27.987191 29664 server_base.cc:1034] running on GCE node
I20260928 16:39:27.988220 29664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:27.989832 29664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:27.993316 29664 hybrid_clock.cc:648] HybridClock initialized: now 1790613567993179 us; error 158 us; skew 500 ppm
Sep 28 16:39:27 dist-test-slave-nqgh krb5kdc[29423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613567, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:28.000798 29664 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:39:28.003244 29664 webserver.cc:460] Webserver started at http://127.25.71.66:40513/ using document root <none> and password file <none>
I20260928 16:39:28.004702 29664 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:28.004869 29664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:28.005327 29664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:28.010039 29664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "1a72b0a37b964109b34021706febb6e2"
format_stamp: "Formatted at 2026-09-28 16:39:27 on dist-test-slave-nqgh"
server_key: "f35958e5b6d7318df2494297a53e5086"
server_key_iv: "46888739082a81f4f33664bfe09d7068"
server_key_version: "encryptionkey@0"
I20260928 16:39:28.011380 29664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "1a72b0a37b964109b34021706febb6e2"
format_stamp: "Formatted at 2026-09-28 16:39:27 on dist-test-slave-nqgh"
server_key: "f35958e5b6d7318df2494297a53e5086"
server_key_iv: "46888739082a81f4f33664bfe09d7068"
server_key_version: "encryptionkey@0"
I20260928 16:39:28.019093 29664 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260928 16:39:28.023068 29679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:28.025168 29664 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:28.025382 29664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "1a72b0a37b964109b34021706febb6e2"
format_stamp: "Formatted at 2026-09-28 16:39:27 on dist-test-slave-nqgh"
server_key: "f35958e5b6d7318df2494297a53e5086"
server_key_iv: "46888739082a81f4f33664bfe09d7068"
server_key_version: "encryptionkey@0"
I20260928 16:39:28.025552 29664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:28.071044 29664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:28.079770 29664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:28.080240 29664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:28.081796 29664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:28.084075 29664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:28.084194 29664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:28.084380 29664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:28.084422 29664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:28.102876 29664 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:43757
I20260928 16:39:28.103003 29796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:43757 every 8 connection(s)
I20260928 16:39:28.105151 29664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:39:28.106278 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 29664
I20260928 16:39:28.106568 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:39:28.107339 25885 external_mini_cluster.cc:1428] Setting key d97372cf9cfd1ba7d86368bd8f147aac
WARNING: no policy specified for kudu/127.25.71.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.67@KRBTEST.COM" created.
Sep 28 16:39:28 dist-test-slave-nqgh krb5kdc[29423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613567, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:28.143798 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:28.110439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:49803 (local address 127.25.71.126:42977)
0928 16:39:28.110874 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:28.110882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:28.113689 (+ 2807us) server_negotiation.cc:407] Connection header received
0928 16:39:28.117142 (+ 3453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:28.117149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:28.117310 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:28.117485 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:28.123052 (+ 5567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:28.124188 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:28.129444 (+ 5256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:28.130008 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:28.130213 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:28.136805 (+ 6592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:28.136810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:28.136812 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:28.136849 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:28.139345 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:28.140317 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:28.140329 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:28.140330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:28.140476 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:28.141158 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:28.141162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:28.141163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:28.141498 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:28.141672 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:28.143016 (+ 1344us) server_negotiation.cc:299] Negotiation successful
0928 16:39:28.143398 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260928 16:39:28.145848 29799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:28.111829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:42977 (local address 127.25.71.66:49803)
0928 16:39:28.113211 (+ 1382us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:28.113346 (+ 135us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:28.116416 (+ 3070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:28.118104 (+ 1688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:28.118139 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:28.119648 (+ 1509us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:28.122323 (+ 2675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:28.122396 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:28.124652 (+ 2256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:28.124661 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:28.128814 (+ 4153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:28.128876 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:28.130421 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:28.130427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:28.131001 (+ 574us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:28.132455 (+ 1454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:28.132513 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:28.136339 (+ 3826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:28.139594 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:28.139602 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:28.139623 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:28.140112 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:28.140634 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:28.140646 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:28.140649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:28.140960 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:28.141679 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:28.141699 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:28.142400 (+ 701us) client_negotiation.cc:769] Sending connection context
0928 16:39:28.142986 (+ 586us) client_negotiation.cc:240] Negotiation successful
0928 16:39:28.143898 (+ 912us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":341,"threads_started":1}
I20260928 16:39:28.148066 29797 heartbeater.cc:347] Connected to a master server at 127.25.71.126:42977
I20260928 16:39:28.148416 29797 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:28.149335 29797 heartbeater.cc:511] Master 127.25.71.126:42977 requested a full tablet report, sending...
I20260928 16:39:28.151814 29468 ts_manager.cc:194] Registered new tserver with Master: 1a72b0a37b964109b34021706febb6e2 (127.25.71.66:43757)
I20260928 16:39:28.153297 29468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:49803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
I20260928 16:39:28.167776 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:42977
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:28.439073 29802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:28.439774 29802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:28.439873 29802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:28.449038 29802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:28.449231 29802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:28.449414 29802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:39:28.462532 29802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:42977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:28.465376 29802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:28.468807 29802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:28.492215 29810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:28.492225 29808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:28.492311 29807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:28.492976 29802 server_base.cc:1034] running on GCE node
I20260928 16:39:28.494207 29802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:28.495889 29802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:28.499320 29802 hybrid_clock.cc:648] HybridClock initialized: now 1790613568499206 us; error 98 us; skew 500 ppm
Sep 28 16:39:28 dist-test-slave-nqgh krb5kdc[29423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613568, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:28.506152 29802 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:39:28.508947 29802 webserver.cc:460] Webserver started at http://127.25.71.67:45755/ using document root <none> and password file <none>
I20260928 16:39:28.510486 29802 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:28.510782 29802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:28.511353 29802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:28.516692 29802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "457089b2aa7046749daf9a36fdf1411f"
format_stamp: "Formatted at 2026-09-28 16:39:28 on dist-test-slave-nqgh"
server_key: "7288a803fb62779c5e7bd50cc912c1f0"
server_key_iv: "45040e26eecb1e018804739007059655"
server_key_version: "encryptionkey@0"
I20260928 16:39:28.518151 29802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "457089b2aa7046749daf9a36fdf1411f"
format_stamp: "Formatted at 2026-09-28 16:39:28 on dist-test-slave-nqgh"
server_key: "7288a803fb62779c5e7bd50cc912c1f0"
server_key_iv: "45040e26eecb1e018804739007059655"
server_key_version: "encryptionkey@0"
I20260928 16:39:28.528338 29802 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.000s
I20260928 16:39:28.535568 29818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:28.539615 29802 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.005s sys 0.000s
I20260928 16:39:28.539981 29802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "457089b2aa7046749daf9a36fdf1411f"
format_stamp: "Formatted at 2026-09-28 16:39:28 on dist-test-slave-nqgh"
server_key: "7288a803fb62779c5e7bd50cc912c1f0"
server_key_iv: "45040e26eecb1e018804739007059655"
server_key_version: "encryptionkey@0"
I20260928 16:39:28.540181 29802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:28.574857 29802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:28.581264 29802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:28.581630 29802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:28.582973 29802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:28.584769 29802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:28.584903 29802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:28.585063 29802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:28.585119 29802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:28.606925 29802 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:37737
I20260928 16:39:28.607137 29933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:37737 every 8 connection(s)
I20260928 16:39:28.609189 29802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:39:28.610590 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 29802
I20260928 16:39:28.610846 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:39:28.611563 25885 external_mini_cluster.cc:1428] Setting key 58a28229d1485db67451ff26e338ebda
Sep 28 16:39:28 dist-test-slave-nqgh krb5kdc[29423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613568, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:28.649299 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:28.614070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:34307 (local address 127.25.71.126:42977)
0928 16:39:28.614326 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:28.614329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:28.617225 (+ 2896us) server_negotiation.cc:407] Connection header received
0928 16:39:28.620732 (+ 3507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:28.620736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:28.620887 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:28.621100 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:28.625286 (+ 4186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:28.626392 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:28.629927 (+ 3535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:28.630383 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:28.630569 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:28.637010 (+ 6441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:28.637024 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:28.637026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:28.637089 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:28.644234 (+ 7145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:28.645473 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:28.645485 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:28.645486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:28.645600 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:28.646277 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:28.646289 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:28.646290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:28.646757 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:28.646927 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:28.648529 (+ 1602us) server_negotiation.cc:299] Negotiation successful
0928 16:39:28.649011 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 16:39:28.651799 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:28.615416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:42977 (local address 127.25.71.67:34307)
0928 16:39:28.616913 (+ 1497us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:28.617047 (+ 134us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:28.620069 (+ 3022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:28.621716 (+ 1647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:28.621743 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:28.623180 (+ 1437us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:28.624723 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:28.624760 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:28.626786 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:28.626791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:28.629551 (+ 2760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:28.629591 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:28.630767 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:28.630779 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:28.631180 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:28.632586 (+ 1406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:28.632616 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:28.636435 (+ 3819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:28.644533 (+ 8098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:28.644551 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:28.644588 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:28.645243 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:28.645769 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:28.645782 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:28.645784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:28.646083 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:28.646931 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:28.646985 (+ 54us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:28.647829 (+ 844us) client_negotiation.cc:769] Sending connection context
0928 16:39:28.648368 (+ 539us) client_negotiation.cc:240] Negotiation successful
0928 16:39:28.649742 (+ 1374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"thread_start_us":377,"threads_started":1}
I20260928 16:39:28.655411 29934 heartbeater.cc:347] Connected to a master server at 127.25.71.126:42977
I20260928 16:39:28.656019 29934 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:28.657510 29934 heartbeater.cc:511] Master 127.25.71.126:42977 requested a full tablet report, sending...
I20260928 16:39:28.659862 29468 ts_manager.cc:194] Registered new tserver with Master: 457089b2aa7046749daf9a36fdf1411f (127.25.71.67:37737)
I20260928 16:39:28.661164 29468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:34307
I20260928 16:39:28.675580 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:39:28 dist-test-slave-nqgh krb5kdc[29423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:39:28 dist-test-slave-nqgh krb5kdc[29423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.65@KRBTEST.COM
I20260928 16:39:28.736168 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:28.709689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54414 (local address 127.25.71.65:43753)
0928 16:39:28.710404 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:28.710423 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:28.710753 (+ 330us) server_negotiation.cc:407] Connection header received
0928 16:39:28.711010 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:28.711020 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:28.711602 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:28.711869 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:28.713453 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:28.715264 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:28.717203 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:28.718330 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:28.718559 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:28.724435 (+ 5876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:28.724471 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:28.724509 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:28.724623 (+ 114us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:28.729016 (+ 4393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:28.730614 (+ 1598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:28.730675 (+ 61us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:28.730677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:28.731005 (+ 328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:28.732101 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:28.732108 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:28.732110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:28.733426 (+ 1316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:28.734101 (+ 675us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:28.735137 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0928 16:39:28.735734 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":249,"threads_started":1}
Sep 28 16:39:28 dist-test-slave-nqgh krb5kdc[29423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:28.758976 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:28.744721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.25.71.126:42977)
0928 16:39:28.745049 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:28.745055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:28.745362 (+ 307us) server_negotiation.cc:407] Connection header received
0928 16:39:28.745654 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:28.745659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:28.745743 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:28.745873 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:28.747445 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:28.748532 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:28.750478 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:28.751333 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:28.751632 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:28.754580 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:28.754584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:28.754587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:28.754657 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:28.756513 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:28.757262 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:28.757265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:28.757266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:28.757390 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:28.757744 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:28.757748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:28.757749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:28.758025 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:28.758160 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:28.758684 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 16:39:28.758814 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
W20260928 16:39:28.759647 29468 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60460
I20260928 16:39:28.760691 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 29530
I20260928 16:39:28.767795 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 29664
I20260928 16:39:28.773888 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 29802
I20260928 16:39:28.779569 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 29436
2026-09-28T16:39:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2710 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:39:28 dist-test-slave-nqgh krb5kdc[29946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:39:28 dist-test-slave-nqgh krb5kdc[29946](info): set up 2 sockets
Sep 28 16:39:28 dist-test-slave-nqgh krb5kdc[29946](info): commencing operation
krb5kdc: starting...
W20260928 16:39:29.773801 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:39:29 dist-test-slave-nqgh krb5kdc[29946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:39:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:39:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:39:29.910712 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:36839
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:36839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:30.048652 29959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:30.049057 29959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:30.049113 29959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:30.053848 29959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:30.053982 29959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:30.054003 29959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:30.054019 29959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:30.054035 29959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:39:30.060678 29959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:36839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:36839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:30.062233 29959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:30.063668 29959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:30.074512 29964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:30.074512 29967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:30.074724 29959 server_base.cc:1034] running on GCE node
W20260928 16:39:30.074980 29965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:30.075933 29959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:30.077564 29959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:30.078791 29959 hybrid_clock.cc:648] HybridClock initialized: now 1790613570078754 us; error 60 us; skew 500 ppm
Sep 28 16:39:30 dist-test-slave-nqgh krb5kdc[29946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613570, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:30.082710 29959 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:39:30.083802 29959 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:30.083892 29959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:30.084156 29959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:30.087071 29959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "9d834740f87940d6ae7303c8d9e4a363"
format_stamp: "Formatted at 2026-09-28 16:39:30 on dist-test-slave-nqgh"
server_key: "6fb9961d693ab33d5145c26312015d05"
server_key_iv: "b120c772e04a4f4439155b51387592a4"
server_key_version: "encryptionkey@0"
I20260928 16:39:30.087810 29959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "9d834740f87940d6ae7303c8d9e4a363"
format_stamp: "Formatted at 2026-09-28 16:39:30 on dist-test-slave-nqgh"
server_key: "6fb9961d693ab33d5145c26312015d05"
server_key_iv: "b120c772e04a4f4439155b51387592a4"
server_key_version: "encryptionkey@0"
I20260928 16:39:30.093014 29959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:39:30.097520 29973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:30.099725 29959 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:39:30.099979 29959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "9d834740f87940d6ae7303c8d9e4a363"
format_stamp: "Formatted at 2026-09-28 16:39:30 on dist-test-slave-nqgh"
server_key: "6fb9961d693ab33d5145c26312015d05"
server_key_iv: "b120c772e04a4f4439155b51387592a4"
server_key_version: "encryptionkey@0"
I20260928 16:39:30.100255 29959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:30.130013 29959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:30.134368 29959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:30.134617 29959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:30.143821 29959 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:36839
I20260928 16:39:30.144136 30026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:36839 every 8 connection(s)
I20260928 16:39:30.144749 29959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:39:30.148221 30027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:30.149788 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 29959
I20260928 16:39:30.149935 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:39:30.150255 25885 external_mini_cluster.cc:1428] Setting key 4593bc37431099177b6fe849382b772f
I20260928 16:39:30.156188 30027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363: Bootstrap starting.
I20260928 16:39:30.159047 30027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:30.159929 30027 log.cc:824] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:39:30 dist-test-slave-nqgh krb5kdc[29946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:30.162678 30027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363: No bootstrap required, opened a new log
I20260928 16:39:30.167271 30027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d834740f87940d6ae7303c8d9e4a363" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 36839 } }
I20260928 16:39:30.167582 30027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:30.167649 30027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d834740f87940d6ae7303c8d9e4a363, State: Initialized, Role: FOLLOWER
I20260928 16:39:30.168172 30027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d834740f87940d6ae7303c8d9e4a363" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 36839 } }
I20260928 16:39:30.168349 30027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:39:30.168433 30027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:39:30.168543 30027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:30.169039 30030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:30.152310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45878 (local address 127.25.71.126:36839)
0928 16:39:30.152896 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:30.152918 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:30.152970 (+ 52us) server_negotiation.cc:407] Connection header received
0928 16:39:30.153903 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:30.153927 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:30.154434 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:30.154916 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:30.155996 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:30.157375 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:30.158750 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:30.159543 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:30.159729 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:30.162958 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:30.162965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:30.162985 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:30.163025 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:30.165271 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:30.165930 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:30.165937 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:30.165951 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:30.166110 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:30.166416 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:30.166419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:30.166421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:30.167048 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:30.167270 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:30.167591 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 16:39:30.167931 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":152,"threads_started":1}
I20260928 16:39:30.170151 30027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d834740f87940d6ae7303c8d9e4a363" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 36839 } }
I20260928 16:39:30.170670 30027 leader_election.cc:302] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d834740f87940d6ae7303c8d9e4a363; no voters:
I20260928 16:39:30.171060 30027 leader_election.cc:288] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:39:30.171159 30032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:30.171360 30032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363 [term 1 LEADER]: Becoming Leader. State: Replica: 9d834740f87940d6ae7303c8d9e4a363, State: Running, Role: LEADER
I20260928 16:39:30.171700 30032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d834740f87940d6ae7303c8d9e4a363" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 36839 } }
I20260928 16:39:30.172559 30027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:30.173935 30033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d834740f87940d6ae7303c8d9e4a363" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d834740f87940d6ae7303c8d9e4a363" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 36839 } } }
I20260928 16:39:30.174175 30033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:30.174768 30034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d834740f87940d6ae7303c8d9e4a363. Latest consensus state: current_term: 1 leader_uuid: "9d834740f87940d6ae7303c8d9e4a363" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d834740f87940d6ae7303c8d9e4a363" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 36839 } } }
I20260928 16:39:30.174963 30034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363 [sys.catalog]: This master's current role is: LEADER
W20260928 16:39:30.178350 30048 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:39:30.178498 30048 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:39:30.178571 30044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:39:30.179589 30044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:39:30.187281 30044 catalog_manager.cc:1347] Generated new cluster ID: 20cc74429a9945eea7b95f9d303f8ba7
I20260928 16:39:30.187399 30044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:39:30.197419 30044 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:39:30.198772 30044 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:39:30.205034 30044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363: Generated new TSK 0
I20260928 16:39:30.206086 30044 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
I20260928 16:39:30.244412 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:36839
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:30.382920 30054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:30.383313 30054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:30.383371 30054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:30.388159 30054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:30.388265 30054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:30.388341 30054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:39:30.394944 30054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:36839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:30.396474 30054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:30.397778 30054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:30.410243 30060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:30.410359 30059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:30.410549 30054 server_base.cc:1034] running on GCE node
W20260928 16:39:30.410386 30062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:30.411384 30054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:30.412423 30054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:30.413779 30054 hybrid_clock.cc:648] HybridClock initialized: now 1790613570413736 us; error 59 us; skew 500 ppm
Sep 28 16:39:30 dist-test-slave-nqgh krb5kdc[29946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613570, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:30.417721 30054 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:39:30.418776 30054 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:30.418870 30054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:30.419147 30054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:30.421725 30054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "a665fa6bfccf4149b16e931eb35681fa"
format_stamp: "Formatted at 2026-09-28 16:39:30 on dist-test-slave-nqgh"
server_key: "1bb64f35816d09eda84bfc62d4c7c91e"
server_key_iv: "fdd125f0ba6e98e71e7983d596b0055b"
server_key_version: "encryptionkey@0"
I20260928 16:39:30.422515 30054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "a665fa6bfccf4149b16e931eb35681fa"
format_stamp: "Formatted at 2026-09-28 16:39:30 on dist-test-slave-nqgh"
server_key: "1bb64f35816d09eda84bfc62d4c7c91e"
server_key_iv: "fdd125f0ba6e98e71e7983d596b0055b"
server_key_version: "encryptionkey@0"
I20260928 16:39:30.427630 30054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 16:39:30.431413 30068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:30.433430 30054 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:30.433621 30054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "a665fa6bfccf4149b16e931eb35681fa"
format_stamp: "Formatted at 2026-09-28 16:39:30 on dist-test-slave-nqgh"
server_key: "1bb64f35816d09eda84bfc62d4c7c91e"
server_key_iv: "fdd125f0ba6e98e71e7983d596b0055b"
server_key_version: "encryptionkey@0"
I20260928 16:39:30.433763 30054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:30.458024 30054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:30.461849 30054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:30.462145 30054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:30.462852 30054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:30.463814 30054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:30.463866 30054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:30.463933 30054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:30.463961 30054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:30.474346 30054 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:42311
I20260928 16:39:30.474406 30183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:42311 every 8 connection(s)
I20260928 16:39:30.475178 30054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:39:30.484167 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 30054
I20260928 16:39:30.484321 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:39:30.484642 25885 external_mini_cluster.cc:1428] Setting key 319c651fab4723c78261d648feede334
Sep 28 16:39:30 dist-test-slave-nqgh krb5kdc[29946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613570, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:30.493182 30030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:30.477797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:44469 (local address 127.25.71.126:36839)
0928 16:39:30.478141 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:30.478148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:30.479810 (+ 1662us) server_negotiation.cc:407] Connection header received
0928 16:39:30.481063 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:30.481067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:30.481134 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:30.481248 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:30.483261 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:30.483867 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:30.485553 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:30.485846 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:30.485971 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:30.489255 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:30.489259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:30.489261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:30.489295 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:30.490770 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:30.491385 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:30.491388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:30.491390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:30.491471 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:30.491823 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:30.491826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:30.491828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:30.492087 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:30.492175 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:30.492814 (+ 639us) server_negotiation.cc:299] Negotiation successful
0928 16:39:30.492974 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
WARNING: no policy specified for kudu/127.25.71.66@KRBTEST.COM; defaulting to no policy
I20260928 16:39:30.494338 30186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:30.478315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:36839 (local address 127.25.71.65:44469)
0928 16:39:30.479522 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:30.479593 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:30.480741 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:30.481592 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:30.481603 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:30.482190 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:30.483001 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:30.483023 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:30.484119 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:30.484124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:30.485404 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:30.485421 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:30.486130 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:30.486135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:30.486421 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:30.487238 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:30.487259 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:30.489001 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:30.490918 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:30.490926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:30.490949 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:30.491271 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:30.491554 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:30.491558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:30.491560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:30.491725 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:30.492190 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:30.492199 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:30.492542 (+ 343us) client_negotiation.cc:769] Sending connection context
0928 16:39:30.492746 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 16:39:30.493188 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":178,"threads_started":1}
I20260928 16:39:30.496290 30184 heartbeater.cc:347] Connected to a master server at 127.25.71.126:36839
I20260928 16:39:30.496721 30184 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.66@KRBTEST.COM" created.
I20260928 16:39:30.497535 30184 heartbeater.cc:511] Master 127.25.71.126:36839 requested a full tablet report, sending...
I20260928 16:39:30.499764 29990 ts_manager.cc:194] Registered new tserver with Master: a665fa6bfccf4149b16e931eb35681fa (127.25.71.65:42311)
I20260928 16:39:30.501330 29990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:44469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
I20260928 16:39:30.515267 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:36839
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:30.642177 30189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:30.642480 30189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:30.642527 30189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:30.647405 30189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:30.647535 30189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:30.647620 30189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:39:30.654903 30189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:36839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:30.656455 30189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:30.657979 30189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:30.671306 30197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:30.671306 30195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:30.671393 30189 server_base.cc:1034] running on GCE node
W20260928 16:39:30.671306 30194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:30.672245 30189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:30.673075 30189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:30.674335 30189 hybrid_clock.cc:648] HybridClock initialized: now 1790613570674292 us; error 61 us; skew 500 ppm
Sep 28 16:39:30 dist-test-slave-nqgh krb5kdc[29946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613570, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:30.677722 30189 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:39:30.678710 30189 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:30.678808 30189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:30.679040 30189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:30.681278 30189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "d23d9bcf8e0648f7bce6f99ed52f33d2"
format_stamp: "Formatted at 2026-09-28 16:39:30 on dist-test-slave-nqgh"
server_key: "4e65f4cf61c4d4385436354e5b2c1526"
server_key_iv: "af8c118e06f2bdf46d574215fd5beec1"
server_key_version: "encryptionkey@0"
I20260928 16:39:30.682062 30189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "d23d9bcf8e0648f7bce6f99ed52f33d2"
format_stamp: "Formatted at 2026-09-28 16:39:30 on dist-test-slave-nqgh"
server_key: "4e65f4cf61c4d4385436354e5b2c1526"
server_key_iv: "af8c118e06f2bdf46d574215fd5beec1"
server_key_version: "encryptionkey@0"
I20260928 16:39:30.687274 30189 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 16:39:30.690477 30203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:30.692317 30189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:39:30.692517 30189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "d23d9bcf8e0648f7bce6f99ed52f33d2"
format_stamp: "Formatted at 2026-09-28 16:39:30 on dist-test-slave-nqgh"
server_key: "4e65f4cf61c4d4385436354e5b2c1526"
server_key_iv: "af8c118e06f2bdf46d574215fd5beec1"
server_key_version: "encryptionkey@0"
I20260928 16:39:30.692651 30189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:30.716451 30189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:30.719738 30189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:30.719936 30189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:30.720544 30189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:30.721468 30189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:30.721520 30189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:30.721576 30189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:30.721603 30189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:30.731979 30189 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:45793
I20260928 16:39:30.732029 30318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:45793 every 8 connection(s)
I20260928 16:39:30.732641 30189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:39:30.733769 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 30189
I20260928 16:39:30.733920 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:39:30.734293 25885 external_mini_cluster.cc:1428] Setting key 644fdee54beefe127e1c1f6471063f0c
WARNING: no policy specified for kudu/127.25.71.67@KRBTEST.COM; defaulting to no policy
Sep 28 16:39:30 dist-test-slave-nqgh krb5kdc[29946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613570, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.67@KRBTEST.COM" created.
I20260928 16:39:30.751784 30030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:30.735942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:52161 (local address 127.25.71.126:36839)
0928 16:39:30.736119 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:30.736125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:30.736755 (+ 630us) server_negotiation.cc:407] Connection header received
0928 16:39:30.738447 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:30.738451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:30.738530 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:30.738613 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:30.740656 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:30.741554 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:30.743562 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:30.743800 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:30.743895 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:30.747038 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:30.747041 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:30.747045 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:30.747073 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:30.748810 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:30.749501 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:30.749506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:30.749507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:30.749589 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:30.750193 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:30.750197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:30.750199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:30.750459 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:30.750560 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:30.751438 (+ 878us) server_negotiation.cc:299] Negotiation successful
0928 16:39:30.751585 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:39:30.753108 30321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:30.735855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:36839 (local address 127.25.71.66:52161)
0928 16:39:30.736562 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:30.736632 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:30.738184 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:30.738912 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:30.738923 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:30.739527 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:30.740380 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:30.740404 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:30.741799 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:30.741806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:30.743404 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:30.743426 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:30.743973 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:30.743977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:30.744202 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:30.744926 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:30.744945 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:30.746811 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:30.748954 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:30.748961 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:30.749009 (+ 48us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:30.749355 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:30.749745 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:30.749749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:30.749751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:30.750040 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:30.750675 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:30.750686 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:30.751086 (+ 400us) client_negotiation.cc:769] Sending connection context
0928 16:39:30.751328 (+ 242us) client_negotiation.cc:240] Negotiation successful
0928 16:39:30.751750 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":164,"threads_started":1}
I20260928 16:39:30.755126 30319 heartbeater.cc:347] Connected to a master server at 127.25.71.126:36839
I20260928 16:39:30.755574 30319 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:30.756377 30319 heartbeater.cc:511] Master 127.25.71.126:36839 requested a full tablet report, sending...
I20260928 16:39:30.758183 29990 ts_manager.cc:194] Registered new tserver with Master: d23d9bcf8e0648f7bce6f99ed52f33d2 (127.25.71.66:45793)
I20260928 16:39:30.759002 29990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:52161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
I20260928 16:39:30.770151 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:36839
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:30.909900 30324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:30.910326 30324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:30.910415 30324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:30.915205 30324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:30.915319 30324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:30.915395 30324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:39:30.922293 30324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:36839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:30.923995 30324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:30.925395 30324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:39:30.937752 30324 server_base.cc:1034] running on GCE node
W20260928 16:39:30.937676 30330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:30.937678 30331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:30.937676 30333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:30.938804 30324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:30.939747 30324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:30.940944 30324 hybrid_clock.cc:648] HybridClock initialized: now 1790613570940915 us; error 59 us; skew 500 ppm
Sep 28 16:39:30 dist-test-slave-nqgh krb5kdc[29946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613570, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:30.944348 30324 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:39:30.945343 30324 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:30.945423 30324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:30.945636 30324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:30.947978 30324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "31c7b1897c0a4e46af1a0f8b88e8ef58"
format_stamp: "Formatted at 2026-09-28 16:39:30 on dist-test-slave-nqgh"
server_key: "eef6062a8d2907d794d715d6f9738ec0"
server_key_iv: "f52f80d8c468bebb7f1edefba321a53b"
server_key_version: "encryptionkey@0"
I20260928 16:39:30.948622 30324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "31c7b1897c0a4e46af1a0f8b88e8ef58"
format_stamp: "Formatted at 2026-09-28 16:39:30 on dist-test-slave-nqgh"
server_key: "eef6062a8d2907d794d715d6f9738ec0"
server_key_iv: "f52f80d8c468bebb7f1edefba321a53b"
server_key_version: "encryptionkey@0"
I20260928 16:39:30.953269 30324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:39:30.956559 30339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:30.958367 30324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:39:30.958561 30324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "31c7b1897c0a4e46af1a0f8b88e8ef58"
format_stamp: "Formatted at 2026-09-28 16:39:30 on dist-test-slave-nqgh"
server_key: "eef6062a8d2907d794d715d6f9738ec0"
server_key_iv: "f52f80d8c468bebb7f1edefba321a53b"
server_key_version: "encryptionkey@0"
I20260928 16:39:30.958760 30324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:30.988242 30324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:30.992282 30324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:30.992486 30324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:30.993110 30324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:30.994042 30324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:30.994092 30324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:30.994145 30324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:30.994174 30324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:31.004562 30324 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:42717
I20260928 16:39:31.004652 30452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:42717 every 8 connection(s)
I20260928 16:39:31.005358 30324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:39:31.009884 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 30324
I20260928 16:39:31.010082 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:39:31.010475 25885 external_mini_cluster.cc:1428] Setting key c4dc2c00a7032dfdbefd3ffcd359a4ea
Sep 28 16:39:31 dist-test-slave-nqgh krb5kdc[29946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613570, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:31.024488 30030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:31.007974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:47639 (local address 127.25.71.126:36839)
0928 16:39:31.008214 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:31.008219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:31.009712 (+ 1493us) server_negotiation.cc:407] Connection header received
0928 16:39:31.011372 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:31.011377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:31.011511 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:31.011630 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:31.013555 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.014234 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.015701 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.016032 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.016140 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:31.019462 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:31.019466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:31.019469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:31.019513 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:31.021494 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:31.022264 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:31.022268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:31.022269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:31.022398 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:31.022948 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:31.022951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:31.022953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:31.023224 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:31.023327 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:31.024094 (+ 767us) server_negotiation.cc:299] Negotiation successful
0928 16:39:31.024267 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260928 16:39:31.025489 30456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:31.008420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:36839 (local address 127.25.71.67:47639)
0928 16:39:31.009398 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:31.009483 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:31.010960 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:31.012107 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:31.012122 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:31.012705 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:31.013357 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:31.013376 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.014412 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.014415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:31.015535 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:31.015549 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.016251 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.016255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:31.016450 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:31.017183 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:31.017203 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:31.019187 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:31.021692 (+ 2505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:31.021699 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:31.021717 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:31.022088 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:31.022534 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:31.022536 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:31.022538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:31.022812 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:31.023366 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:31.023373 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:31.023700 (+ 327us) client_negotiation.cc:769] Sending connection context
0928 16:39:31.023927 (+ 227us) client_negotiation.cc:240] Negotiation successful
0928 16:39:31.024394 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":145,"threads_started":1}
I20260928 16:39:31.027086 30454 heartbeater.cc:347] Connected to a master server at 127.25.71.126:36839
I20260928 16:39:31.027400 30454 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:31.028016 30454 heartbeater.cc:511] Master 127.25.71.126:36839 requested a full tablet report, sending...
I20260928 16:39:31.029764 29990 ts_manager.cc:194] Registered new tserver with Master: 31c7b1897c0a4e46af1a0f8b88e8ef58 (127.25.71.67:42717)
I20260928 16:39:31.030550 29990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:47639
I20260928 16:39:31.039718 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:39:31.051357 30030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:31.042333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45892 (local address 127.25.71.126:36839)
0928 16:39:31.042504 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:31.042508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:31.042563 (+ 55us) server_negotiation.cc:407] Connection header received
0928 16:39:31.042721 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:31.042723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:31.042785 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:31.042945 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:31.043777 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.044396 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.045393 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.045663 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.045775 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:31.047376 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:31.047380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:31.047382 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:31.047417 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:31.049052 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:31.049701 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:31.049705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:31.049706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:31.049812 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:31.050217 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:31.050220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:31.050221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:31.050524 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:31.050720 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:31.051071 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 16:39:31.051208 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:39:31.056227 29990 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:39:31.061517 29990 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:39:31.081040 30468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:31.074730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53346 (local address 127.25.71.66:45793)
0928 16:39:31.076054 (+ 1324us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:31.076065 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:31.076088 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:39:31.076167 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:31.076171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:31.076350 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:31.076526 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:31.076826 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.077825 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.079257 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.080114 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.080248 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:31.080572 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:31.080649 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 16:39:31.080760 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1199,"thread_start_us":171,"threads_started":1}
I20260928 16:39:31.081035 30470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:31.074517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:45793 (local address 127.0.0.1:53346)
0928 16:39:31.075384 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:31.075400 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:31.075551 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:31.076536 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:31.076540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:31.076570 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:31.076682 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:31.076690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.077971 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.077975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:31.079123 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:31.079134 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.080286 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.080290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:31.080422 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:31.080470 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 16:39:31.080604 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 16:39:31.080798 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":4,"thread_start_us":686,"threads_started":1}
I20260928 16:39:31.081058 30467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:31.073413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:42717 (local address 127.0.0.1:44174)
0928 16:39:31.074733 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:31.074801 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:31.074960 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:31.075874 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:31.075878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:31.075901 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:31.076062 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:31.076072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.077567 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.077570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:31.079145 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:31.079157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.080672 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.080676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:31.080770 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:31.080795 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 16:39:31.080868 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 16:39:31.080939 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1088,"mutex_wait_us":96,"spinlock_wait_cycles":4608,"thread_start_us":120,"threads_started":1}
I20260928 16:39:31.081771 30465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:31.074288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:42311 (local address 127.0.0.1:60934)
0928 16:39:31.074733 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:31.074801 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:31.074960 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:31.075746 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:31.075754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:31.075781 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:31.076017 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:31.076029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.077364 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.077368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:31.079661 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:31.079672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.081234 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.081239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:31.081370 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:31.081404 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 16:39:31.081487 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 16:39:31.081601 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":80,"spinlock_wait_cycles":7808,"thread_start_us":134,"threads_started":1}
I20260928 16:39:31.081809 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:31.073745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44174 (local address 127.25.71.67:42717)
0928 16:39:31.074295 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:31.074368 (+ 73us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:31.075122 (+ 754us) server_negotiation.cc:407] Connection header received
0928 16:39:31.075233 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:31.075238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:31.075486 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:31.075654 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:31.076202 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.077389 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.079365 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.080253 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.080386 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:31.080600 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:31.081141 (+ 541us) server_negotiation.cc:299] Negotiation successful
0928 16:39:31.081606 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":263,"threads_started":1}
I20260928 16:39:31.082280 30469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:31.074816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60934 (local address 127.25.71.65:42311)
0928 16:39:31.075310 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:31.075317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:31.075338 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:39:31.075404 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:31.075408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:31.075553 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:31.075714 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:31.076197 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.077237 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.079842 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.080844 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.080981 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:31.081144 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:31.081975 (+ 831us) server_negotiation.cc:299] Negotiation successful
0928 16:39:31.082130 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":101,"threads_started":1}
I20260928 16:39:31.085274 30116 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9d3634257e43ec920455076cd7ab2a (DEFAULT_TABLE table=test-table [id=50c821929bb2417a9a9ce8a2a0df9f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:31.085484 30387 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9d3634257e43ec920455076cd7ab2a (DEFAULT_TABLE table=test-table [id=50c821929bb2417a9a9ce8a2a0df9f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:31.085484 30251 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9d3634257e43ec920455076cd7ab2a (DEFAULT_TABLE table=test-table [id=50c821929bb2417a9a9ce8a2a0df9f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:31.086475 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b9d3634257e43ec920455076cd7ab2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:31.086964 30251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b9d3634257e43ec920455076cd7ab2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:31.086977 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b9d3634257e43ec920455076cd7ab2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:31.095654 30471 tablet_bootstrap.cc:492] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa: Bootstrap starting.
I20260928 16:39:31.097352 30472 tablet_bootstrap.cc:492] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58: Bootstrap starting.
I20260928 16:39:31.097352 30473 tablet_bootstrap.cc:492] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2: Bootstrap starting.
I20260928 16:39:31.098451 30471 tablet_bootstrap.cc:654] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:31.099635 30471 log.cc:824] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:31.100574 30472 tablet_bootstrap.cc:654] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:31.101022 30473 tablet_bootstrap.cc:654] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:31.102015 30471 tablet_bootstrap.cc:492] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa: No bootstrap required, opened a new log
I20260928 16:39:31.102262 30471 ts_tablet_manager.cc:1397] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 16:39:31.102236 30473 log.cc:824] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:31.102237 30472 log.cc:824] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:31.105556 30472 tablet_bootstrap.cc:492] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58: No bootstrap required, opened a new log
I20260928 16:39:31.105525 30473 tablet_bootstrap.cc:492] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2: No bootstrap required, opened a new log
I20260928 16:39:31.105844 30472 ts_tablet_manager.cc:1397] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260928 16:39:31.105844 30473 ts_tablet_manager.cc:1397] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 16:39:31.107944 30471 raft_consensus.cc:348] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c7b1897c0a4e46af1a0f8b88e8ef58" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 42717 } } peers { permanent_uuid: "a665fa6bfccf4149b16e931eb35681fa" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 42311 } } peers { permanent_uuid: "d23d9bcf8e0648f7bce6f99ed52f33d2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 45793 } }
I20260928 16:39:31.108237 30471 raft_consensus.cc:374] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:31.108291 30471 raft_consensus.cc:729] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a665fa6bfccf4149b16e931eb35681fa, State: Initialized, Role: FOLLOWER
I20260928 16:39:31.108779 30471 consensus_queue.cc:260] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c7b1897c0a4e46af1a0f8b88e8ef58" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 42717 } } peers { permanent_uuid: "a665fa6bfccf4149b16e931eb35681fa" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 42311 } } peers { permanent_uuid: "d23d9bcf8e0648f7bce6f99ed52f33d2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 45793 } }
I20260928 16:39:31.109787 30184 heartbeater.cc:503] Master 127.25.71.126:36839 was elected leader, sending a full tablet report...
I20260928 16:39:31.110054 30471 ts_tablet_manager.cc:1428] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260928 16:39:31.111020 30473 raft_consensus.cc:348] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c7b1897c0a4e46af1a0f8b88e8ef58" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 42717 } } peers { permanent_uuid: "a665fa6bfccf4149b16e931eb35681fa" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 42311 } } peers { permanent_uuid: "d23d9bcf8e0648f7bce6f99ed52f33d2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 45793 } }
I20260928 16:39:31.111477 30473 raft_consensus.cc:374] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:31.111552 30473 raft_consensus.cc:729] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23d9bcf8e0648f7bce6f99ed52f33d2, State: Initialized, Role: FOLLOWER
I20260928 16:39:31.112128 30473 consensus_queue.cc:260] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c7b1897c0a4e46af1a0f8b88e8ef58" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 42717 } } peers { permanent_uuid: "a665fa6bfccf4149b16e931eb35681fa" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 42311 } } peers { permanent_uuid: "d23d9bcf8e0648f7bce6f99ed52f33d2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 45793 } }
I20260928 16:39:31.113086 30319 heartbeater.cc:503] Master 127.25.71.126:36839 was elected leader, sending a full tablet report...
I20260928 16:39:31.113440 30473 ts_tablet_manager.cc:1428] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260928 16:39:31.113970 30472 raft_consensus.cc:348] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c7b1897c0a4e46af1a0f8b88e8ef58" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 42717 } } peers { permanent_uuid: "a665fa6bfccf4149b16e931eb35681fa" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 42311 } } peers { permanent_uuid: "d23d9bcf8e0648f7bce6f99ed52f33d2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 45793 } }
I20260928 16:39:31.114357 30472 raft_consensus.cc:374] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:31.114416 30472 raft_consensus.cc:729] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c7b1897c0a4e46af1a0f8b88e8ef58, State: Initialized, Role: FOLLOWER
I20260928 16:39:31.114956 30472 consensus_queue.cc:260] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c7b1897c0a4e46af1a0f8b88e8ef58" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 42717 } } peers { permanent_uuid: "a665fa6bfccf4149b16e931eb35681fa" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 42311 } } peers { permanent_uuid: "d23d9bcf8e0648f7bce6f99ed52f33d2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 45793 } }
I20260928 16:39:31.116261 30472 ts_tablet_manager.cc:1428] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20260928 16:39:31.116657 30454 heartbeater.cc:503] Master 127.25.71.126:36839 was elected leader, sending a full tablet report...
W20260928 16:39:31.226919 30185 tablet.cc:2369] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:39:31.234043 30320 tablet.cc:2369] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:39:31.256628 30455 tablet.cc:2369] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:39:31.284713 30477 raft_consensus.cc:482] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:39:31.285058 30477 raft_consensus.cc:504] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c7b1897c0a4e46af1a0f8b88e8ef58" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 42717 } } peers { permanent_uuid: "a665fa6bfccf4149b16e931eb35681fa" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 42311 } } peers { permanent_uuid: "d23d9bcf8e0648f7bce6f99ed52f33d2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 45793 } }
I20260928 16:39:31.286392 30477 leader_election.cc:288] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31c7b1897c0a4e46af1a0f8b88e8ef58 (127.25.71.67:42717), d23d9bcf8e0648f7bce6f99ed52f33d2 (127.25.71.66:45793)
I20260928 16:39:31.291179 30480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:31.286853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:45793 (local address 127.25.71.65:37263)
0928 16:39:31.287335 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:31.287353 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:31.287497 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:31.287794 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:31.287805 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:31.287832 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:31.288033 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:31.288039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.288803 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.288807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:31.289816 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:31.289827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.290579 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.290584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:31.290780 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:31.290816 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 16:39:31.290899 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 16:39:31.290983 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":4,"spinlock_wait_cycles":26496,"thread_start_us":145,"threads_started":1}
I20260928 16:39:31.291227 30468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:31.286953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:37263 (local address 127.25.71.66:45793)
0928 16:39:31.287138 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:31.287153 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:31.287472 (+ 319us) server_negotiation.cc:407] Connection header received
0928 16:39:31.287624 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:31.287628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:31.287690 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:31.287762 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:31.288149 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.288681 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.289971 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.290436 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.290550 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:31.290596 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:31.290977 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 16:39:31.291074 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 16:39:31.291471 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:31.287008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:44189 (local address 127.25.71.67:42717)
0928 16:39:31.287160 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:31.287166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:31.287530 (+ 364us) server_negotiation.cc:407] Connection header received
0928 16:39:31.287692 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:31.287695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:31.287751 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:31.287834 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:31.288270 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.288860 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.290122 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.290608 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.290787 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:31.290845 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:31.291236 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 16:39:31.291337 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:39:31.291482 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:31.286901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:42717 (local address 127.25.71.65:44189)
0928 16:39:31.287387 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:31.287401 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:31.287560 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:31.287878 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:31.287881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:31.287927 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:31.288079 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:31.288086 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.288987 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.288990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:31.289982 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:31.289995 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.290855 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.290859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:31.291062 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:31.291096 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 16:39:31.291197 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 16:39:31.291298 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":1,"spinlock_wait_cycles":1408,"thread_start_us":206,"threads_started":1}
I20260928 16:39:31.292346 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9d3634257e43ec920455076cd7ab2a" candidate_uuid: "a665fa6bfccf4149b16e931eb35681fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31c7b1897c0a4e46af1a0f8b88e8ef58" is_pre_election: true
I20260928 16:39:31.292399 30272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9d3634257e43ec920455076cd7ab2a" candidate_uuid: "a665fa6bfccf4149b16e931eb35681fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d23d9bcf8e0648f7bce6f99ed52f33d2" is_pre_election: true
I20260928 16:39:31.292716 30407 raft_consensus.cc:2454] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a665fa6bfccf4149b16e931eb35681fa in term 0.
I20260928 16:39:31.292850 30272 raft_consensus.cc:2454] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a665fa6bfccf4149b16e931eb35681fa in term 0.
I20260928 16:39:31.293339 30071 leader_election.cc:302] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31c7b1897c0a4e46af1a0f8b88e8ef58, a665fa6bfccf4149b16e931eb35681fa; no voters:
I20260928 16:39:31.293691 30477 raft_consensus.cc:2781] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:39:31.293813 30477 raft_consensus.cc:482] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:39:31.293870 30477 raft_consensus.cc:3037] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:31.295315 30477 raft_consensus.cc:504] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c7b1897c0a4e46af1a0f8b88e8ef58" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 42717 } } peers { permanent_uuid: "a665fa6bfccf4149b16e931eb35681fa" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 42311 } } peers { permanent_uuid: "d23d9bcf8e0648f7bce6f99ed52f33d2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 45793 } }
I20260928 16:39:31.295789 30477 leader_election.cc:288] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa [CANDIDATE]: Term 1 election: Requested vote from peers 31c7b1897c0a4e46af1a0f8b88e8ef58 (127.25.71.67:42717), d23d9bcf8e0648f7bce6f99ed52f33d2 (127.25.71.66:45793)
I20260928 16:39:31.296464 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9d3634257e43ec920455076cd7ab2a" candidate_uuid: "a665fa6bfccf4149b16e931eb35681fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31c7b1897c0a4e46af1a0f8b88e8ef58"
I20260928 16:39:31.296465 30272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9d3634257e43ec920455076cd7ab2a" candidate_uuid: "a665fa6bfccf4149b16e931eb35681fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d23d9bcf8e0648f7bce6f99ed52f33d2"
I20260928 16:39:31.296641 30407 raft_consensus.cc:3037] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:31.296662 30272 raft_consensus.cc:3037] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:31.298085 30407 raft_consensus.cc:2454] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a665fa6bfccf4149b16e931eb35681fa in term 1.
I20260928 16:39:31.298106 30272 raft_consensus.cc:2454] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a665fa6bfccf4149b16e931eb35681fa in term 1.
I20260928 16:39:31.298575 30071 leader_election.cc:302] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 31c7b1897c0a4e46af1a0f8b88e8ef58, a665fa6bfccf4149b16e931eb35681fa, d23d9bcf8e0648f7bce6f99ed52f33d2; no voters:
I20260928 16:39:31.298872 30477 raft_consensus.cc:2781] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:31.299201 30477 raft_consensus.cc:686] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa [term 1 LEADER]: Becoming Leader. State: Replica: a665fa6bfccf4149b16e931eb35681fa, State: Running, Role: LEADER
I20260928 16:39:31.299654 30477 consensus_queue.cc:237] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c7b1897c0a4e46af1a0f8b88e8ef58" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 42717 } } peers { permanent_uuid: "a665fa6bfccf4149b16e931eb35681fa" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 42311 } } peers { permanent_uuid: "d23d9bcf8e0648f7bce6f99ed52f33d2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 45793 } }
I20260928 16:39:31.303961 29990 catalog_manager.cc:5495] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa reported cstate change: term changed from 0 to 1, leader changed from <none> to a665fa6bfccf4149b16e931eb35681fa (127.25.71.65). New cstate: current_term: 1 leader_uuid: "a665fa6bfccf4149b16e931eb35681fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c7b1897c0a4e46af1a0f8b88e8ef58" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 42717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a665fa6bfccf4149b16e931eb35681fa" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 42311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d23d9bcf8e0648f7bce6f99ed52f33d2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 45793 } health_report { overall_health: UNKNOWN } } }
I20260928 16:39:31.327163 30469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:31.322229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60948 (local address 127.25.71.65:42311)
0928 16:39:31.322360 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:31.322364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:31.322391 (+ 27us) server_negotiation.cc:407] Connection header received
0928 16:39:31.322663 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:31.322667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:31.322852 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:31.323000 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:31.323405 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.324052 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.325184 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:31.325462 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:31.325583 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:31.326101 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:31.326606 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:31.326794 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:31.326941 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 16:39:31.327031 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:39:31.335017 30272 raft_consensus.cc:1261] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2 [term 1 FOLLOWER]: Refusing update from remote peer a665fa6bfccf4149b16e931eb35681fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:31.335280 30407 raft_consensus.cc:1261] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58 [term 1 FOLLOWER]: Refusing update from remote peer a665fa6bfccf4149b16e931eb35681fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:31.336066 30477 consensus_queue.cc:1035] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23d9bcf8e0648f7bce6f99ed52f33d2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 45793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:31.336342 30482 consensus_queue.cc:1035] T 7b9d3634257e43ec920455076cd7ab2a P a665fa6bfccf4149b16e931eb35681fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "31c7b1897c0a4e46af1a0f8b88e8ef58" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 42717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:31.347430 30486 mvcc.cc:205] Tried to move back new op lower bound from 7334353188178575360 to 7334353188046815232. Current Snapshot: MvccSnapshot[applied={T|T < 7334353188178575360}]
I20260928 16:39:31.347891 30487 mvcc.cc:205] Tried to move back new op lower bound from 7334353188178575360 to 7334353188046815232. Current Snapshot: MvccSnapshot[applied={T|T < 7334353188178575360}]
I20260928 16:39:31.360024 29985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:31.360404 29985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:31.364416 29985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d834740f87940d6ae7303c8d9e4a363: Sending DeleteTablet for 3 replicas of tablet 7b9d3634257e43ec920455076cd7ab2a
I20260928 16:39:31.365674 30251 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b9d3634257e43ec920455076cd7ab2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:31 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:53346
I20260928 16:39:31.365720 30116 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b9d3634257e43ec920455076cd7ab2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:31 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:60934
I20260928 16:39:31.366386 30495 tablet_replica.cc:330] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2: stopping tablet replica
I20260928 16:39:31.366447 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 30054
I20260928 16:39:31.366861 30387 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b9d3634257e43ec920455076cd7ab2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:31 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:44174
I20260928 16:39:31.368047 30495 raft_consensus.cc:2229] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:31.368582 30495 raft_consensus.cc:2258] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:31.368572 30497 tablet_replica.cc:330] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58: stopping tablet replica
I20260928 16:39:31.369005 30497 raft_consensus.cc:2229] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:31.369443 30497 raft_consensus.cc:2258] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:31.370499 30495 ts_tablet_manager.cc:1905] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:31.371448 30497 ts_tablet_manager.cc:1905] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:31.373723 30495 ts_tablet_manager.cc:1918] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:39:31.373855 30495 log.cc:1196] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/7b9d3634257e43ec920455076cd7ab2a
I20260928 16:39:31.374235 30495 ts_tablet_manager.cc:1939] T 7b9d3634257e43ec920455076cd7ab2a P d23d9bcf8e0648f7bce6f99ed52f33d2: Deleting consensus metadata
I20260928 16:39:31.375000 30497 ts_tablet_manager.cc:1918] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260928 16:39:31.375077 29976 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.71.65:42311 (error 108)
I20260928 16:39:31.375149 30497 log.cc:1196] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/7b9d3634257e43ec920455076cd7ab2a
I20260928 16:39:31.375674 30497 ts_tablet_manager.cc:1939] T 7b9d3634257e43ec920455076cd7ab2a P 31c7b1897c0a4e46af1a0f8b88e8ef58: Deleting consensus metadata
I20260928 16:39:31.376176 29974 catalog_manager.cc:4841] TS d23d9bcf8e0648f7bce6f99ed52f33d2 (127.25.71.66:45793): tablet 7b9d3634257e43ec920455076cd7ab2a (table test-table [id=50c821929bb2417a9a9ce8a2a0df9f2b]) successfully deleted
I20260928 16:39:31.376376 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 30189
I20260928 16:39:31.376767 30470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:31.376430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:42311 (local address 127.0.0.1:60954)
0928 16:39:31.376584 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:31.376669 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.65:42311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 16:39:31.377429 29976 catalog_manager.cc:4584] TS a665fa6bfccf4149b16e931eb35681fa (127.25.71.65:42311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7b9d3634257e43ec920455076cd7ab2a: Network error: Client connection negotiation failed: client connection to 127.25.71.65:42311: connect: Connection refused (error 111)
I20260928 16:39:31.378912 29976 catalog_manager.cc:4841] TS 31c7b1897c0a4e46af1a0f8b88e8ef58 (127.25.71.67:42717): tablet 7b9d3634257e43ec920455076cd7ab2a (table test-table [id=50c821929bb2417a9a9ce8a2a0df9f2b]) successfully deleted
I20260928 16:39:31.383411 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 30324
I20260928 16:39:31.389346 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 29959
2026-09-28T16:39:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2609 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T16:39:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:39:31Z Disabled control of system clock
I20260928 16:39:31.423833 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:46161
--webserver_interface=127.25.71.126
--webserver_port=0
--builtin_ntp_servers=127.25.71.84:42123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:46161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 16:39:31.564934 30502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:31.565363 30502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:31.565428 30502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:31.570144 30502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:31.570263 30502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:31.570286 30502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:31.570310 30502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:31.570330 30502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:39:31.576267 30502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:42123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:46161
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:46161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:31.577952 30502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:31.579972 30502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:31.591653 30510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:31.591642 30507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:31.591642 30508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:31.591799 30502 server_base.cc:1034] running on GCE node
I20260928 16:39:31.592767 30502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:31.594103 30502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:31.595355 30502 hybrid_clock.cc:648] HybridClock initialized: now 1790613571595322 us; error 59 us; skew 500 ppm
I20260928 16:39:31.597759 30502 webserver.cc:460] Webserver started at http://127.25.71.126:36245/ using document root <none> and password file <none>
I20260928 16:39:31.598454 30502 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:31.598531 30502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:31.598778 30502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:31.601071 30502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "9da726416281415e80afe3363f105f9b"
format_stamp: "Formatted at 2026-09-28 16:39:31 on dist-test-slave-nqgh"
server_key: "e492203612ff7254dca27046a30759dc"
server_key_iv: "ccf2cfd9fe25ab0f54deaeec03dfa14b"
server_key_version: "encryptionkey@0"
I20260928 16:39:31.601732 30502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "9da726416281415e80afe3363f105f9b"
format_stamp: "Formatted at 2026-09-28 16:39:31 on dist-test-slave-nqgh"
server_key: "e492203612ff7254dca27046a30759dc"
server_key_iv: "ccf2cfd9fe25ab0f54deaeec03dfa14b"
server_key_version: "encryptionkey@0"
I20260928 16:39:31.606233 30502 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 16:39:31.609673 30516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:31.611673 30502 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:39:31.611864 30502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "9da726416281415e80afe3363f105f9b"
format_stamp: "Formatted at 2026-09-28 16:39:31 on dist-test-slave-nqgh"
server_key: "e492203612ff7254dca27046a30759dc"
server_key_iv: "ccf2cfd9fe25ab0f54deaeec03dfa14b"
server_key_version: "encryptionkey@0"
I20260928 16:39:31.612037 30502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:31.617327 30502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:31.618094 30502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:31.618284 30502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:31.626104 30502 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:46161
I20260928 16:39:31.626189 30568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:46161 every 8 connection(s)
I20260928 16:39:31.627303 30502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:39:31.632176 30569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:31.632510 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 30502
I20260928 16:39:31.632642 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:39:31.632949 25885 external_mini_cluster.cc:1428] Setting key ceb80a1c38d5587ef6885a6c892d73f6
I20260928 16:39:31.641438 30572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:31.634911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.25.71.126:46161)
0928 16:39:31.635838 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:31.635860 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:31.635914 (+ 54us) server_negotiation.cc:407] Connection header received
0928 16:39:31.637019 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:31.637047 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:31.637605 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:31.638058 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:31.639436 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:31.639443 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:31.639463 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:39:31.639479 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:31.639570 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:39:31.639593 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:31.639766 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:31.639900 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 16:39:31.640348 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":180,"threads_started":1}
I20260928 16:39:31.642217 30569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b: Bootstrap starting.
I20260928 16:39:31.645047 30569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:31.645885 30569 log.cc:824] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:31.648605 30569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b: No bootstrap required, opened a new log
I20260928 16:39:31.652817 30569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da726416281415e80afe3363f105f9b" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 46161 } }
I20260928 16:39:31.653144 30569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:31.653182 30569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da726416281415e80afe3363f105f9b, State: Initialized, Role: FOLLOWER
I20260928 16:39:31.653692 30569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da726416281415e80afe3363f105f9b" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 46161 } }
I20260928 16:39:31.653887 30569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:39:31.653955 30569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:39:31.654070 30569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:31.655429 30569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da726416281415e80afe3363f105f9b" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 46161 } }
I20260928 16:39:31.656104 30569 leader_election.cc:302] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9da726416281415e80afe3363f105f9b; no voters:
I20260928 16:39:31.656700 30569 leader_election.cc:288] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:39:31.657001 30574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:31.657308 30574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b [term 1 LEADER]: Becoming Leader. State: Replica: 9da726416281415e80afe3363f105f9b, State: Running, Role: LEADER
I20260928 16:39:31.657770 30574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da726416281415e80afe3363f105f9b" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 46161 } }
I20260928 16:39:31.658416 30569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:31.660486 30575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9da726416281415e80afe3363f105f9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da726416281415e80afe3363f105f9b" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 46161 } } }
I20260928 16:39:31.660686 30575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:31.660985 30576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9da726416281415e80afe3363f105f9b. Latest consensus state: current_term: 1 leader_uuid: "9da726416281415e80afe3363f105f9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da726416281415e80afe3363f105f9b" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 46161 } } }
I20260928 16:39:31.661124 30588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:39:31.661458 30576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:31.664713 30588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:39:31.672547 30588 catalog_manager.cc:1347] Generated new cluster ID: 630a7736ec5f4dd19f1abf67075ab9a7
I20260928 16:39:31.672653 30588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:39:31.685009 30588 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:39:31.685606 30588 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:39:31.699566 30588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b: Generated new TSK 0
I20260928 16:39:31.700876 30588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:39:31.713868 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:46161
--builtin_ntp_servers=127.25.71.84:42123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 16:39:31.859164 30593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:31.859529 30593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:31.859575 30593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:31.864219 30593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:31.864322 30593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:31.864387 30593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:39:31.869724 30593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:42123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:46161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:31.871117 30593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:31.872924 30593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:31.884186 30599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:31.884485 30601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:31.884941 30598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:31.884975 30593 server_base.cc:1034] running on GCE node
I20260928 16:39:31.885757 30593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:31.886746 30593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:31.887990 30593 hybrid_clock.cc:648] HybridClock initialized: now 1790613571887951 us; error 63 us; skew 500 ppm
I20260928 16:39:31.890931 30593 webserver.cc:460] Webserver started at http://127.25.71.65:43633/ using document root <none> and password file <none>
I20260928 16:39:31.891772 30593 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:31.891865 30593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:31.892122 30593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:31.894996 30593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "c66bbe8691004354b79c4f3e62b27af4"
format_stamp: "Formatted at 2026-09-28 16:39:31 on dist-test-slave-nqgh"
server_key: "1ce708d0464fdaae8e9cdcb77c13406b"
server_key_iv: "02b8104bed29b854907d9a24ae4e22c9"
server_key_version: "encryptionkey@0"
I20260928 16:39:31.895707 30593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "c66bbe8691004354b79c4f3e62b27af4"
format_stamp: "Formatted at 2026-09-28 16:39:31 on dist-test-slave-nqgh"
server_key: "1ce708d0464fdaae8e9cdcb77c13406b"
server_key_iv: "02b8104bed29b854907d9a24ae4e22c9"
server_key_version: "encryptionkey@0"
I20260928 16:39:31.900669 30593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:39:31.904610 30607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:31.906702 30593 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:31.906913 30593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "c66bbe8691004354b79c4f3e62b27af4"
format_stamp: "Formatted at 2026-09-28 16:39:31 on dist-test-slave-nqgh"
server_key: "1ce708d0464fdaae8e9cdcb77c13406b"
server_key_iv: "02b8104bed29b854907d9a24ae4e22c9"
server_key_version: "encryptionkey@0"
I20260928 16:39:31.907112 30593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:31.913179 30593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:31.914072 30593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:31.914284 30593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:31.915033 30593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:31.916289 30593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:31.916343 30593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:31.916407 30593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:31.916431 30593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:31.928025 30593 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:35755
I20260928 16:39:31.928114 30720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:35755 every 8 connection(s)
I20260928 16:39:31.929312 30593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:39:31.933451 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 30593
I20260928 16:39:31.933598 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:39:31.933934 25885 external_mini_cluster.cc:1428] Setting key 36cd22fa6c65f084a4b6f69d56396a41
I20260928 16:39:31.936831 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:46161
--builtin_ntp_servers=127.25.71.84:42123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 16:39:31.938002 30572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:31.931967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:49565 (local address 127.25.71.126:46161)
0928 16:39:31.932193 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:31.932198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:31.933919 (+ 1721us) server_negotiation.cc:407] Connection header received
0928 16:39:31.935735 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:31.935742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:31.935786 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:31.935912 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:31.937474 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:31.937478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:31.937480 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:39:31.937497 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:31.937530 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:39:31.937540 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:31.937650 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:31.937801 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 16:39:31.937854 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260928 16:39:31.939182 30723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:31.932490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:46161 (local address 127.25.71.65:49565)
0928 16:39:31.933600 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:31.933680 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:31.935443 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:31.936153 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:31.936163 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:31.936299 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:31.937090 (+ 791us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:39:31.937092 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:31.937154 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:39:31.937258 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:31.937642 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:31.937650 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:31.937656 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:39:31.937870 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 16:39:31.938182 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":147,"threads_started":1}
I20260928 16:39:31.941330 30721 heartbeater.cc:347] Connected to a master server at 127.25.71.126:46161
I20260928 16:39:31.942090 30721 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:31.942762 30721 heartbeater.cc:511] Master 127.25.71.126:46161 requested a full tablet report, sending...
I20260928 16:39:31.945413 30533 ts_manager.cc:194] Registered new tserver with Master: c66bbe8691004354b79c4f3e62b27af4 (127.25.71.65:35755)
W20260928 16:39:32.077566 30724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:32.078022 30724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:32.078070 30724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:32.083429 30724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:32.083551 30724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:32.083623 30724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:39:32.089707 30724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:42123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:46161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:32.091348 30724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:32.093195 30724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:32.106066 30732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:32.106148 30730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:32.106055 30729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:32.106405 30724 server_base.cc:1034] running on GCE node
I20260928 16:39:32.107448 30724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:32.108497 30724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:32.109745 30724 hybrid_clock.cc:648] HybridClock initialized: now 1790613572109715 us; error 53 us; skew 500 ppm
I20260928 16:39:32.112974 30724 webserver.cc:460] Webserver started at http://127.25.71.66:39073/ using document root <none> and password file <none>
I20260928 16:39:32.114022 30724 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:32.114132 30724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:32.114441 30724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:32.117859 30724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "4ec8b760dafa4f3893190714d50e7baa"
format_stamp: "Formatted at 2026-09-28 16:39:32 on dist-test-slave-nqgh"
server_key: "743fcaa09ea3efd287f0ac7e0bfec27e"
server_key_iv: "706e59c3adaddc1123bbe2915c59871e"
server_key_version: "encryptionkey@0"
I20260928 16:39:32.119127 30724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "4ec8b760dafa4f3893190714d50e7baa"
format_stamp: "Formatted at 2026-09-28 16:39:32 on dist-test-slave-nqgh"
server_key: "743fcaa09ea3efd287f0ac7e0bfec27e"
server_key_iv: "706e59c3adaddc1123bbe2915c59871e"
server_key_version: "encryptionkey@0"
I20260928 16:39:32.125205 30724 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:39:32.129738 30738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:32.132124 30724 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:39:32.132362 30724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "4ec8b760dafa4f3893190714d50e7baa"
format_stamp: "Formatted at 2026-09-28 16:39:32 on dist-test-slave-nqgh"
server_key: "743fcaa09ea3efd287f0ac7e0bfec27e"
server_key_iv: "706e59c3adaddc1123bbe2915c59871e"
server_key_version: "encryptionkey@0"
I20260928 16:39:32.132575 30724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:32.139164 30724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:32.140223 30724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:32.140476 30724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:32.141369 30724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:32.142877 30724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:32.142959 30724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:32.143023 30724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:32.143049 30724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:32.155448 30724 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:35403
I20260928 16:39:32.155503 30853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:35403 every 8 connection(s)
I20260928 16:39:32.156785 30724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:39:32.165689 30572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:32.159837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:59649 (local address 127.25.71.126:46161)
0928 16:39:32.160020 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:32.160025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:32.161933 (+ 1908us) server_negotiation.cc:407] Connection header received
0928 16:39:32.163546 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:32.163550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:32.163607 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:32.163690 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:32.165082 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:32.165086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:32.165090 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:39:32.165106 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:32.165128 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:39:32.165137 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:32.165224 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:32.165351 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 16:39:32.165407 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:39:32.166941 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 30724
I20260928 16:39:32.167052 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:32.160496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:46161 (local address 127.25.71.66:59649)
0928 16:39:32.161725 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:32.161808 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:32.163283 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:32.163917 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:32.163926 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:32.164040 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:32.164710 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:39:32.164712 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:32.164756 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:39:32.164835 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:32.165210 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:32.165219 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:32.165225 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:39:32.165406 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 16:39:32.165778 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":173,"threads_started":1}
I20260928 16:39:32.167112 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:39:32.167506 25885 external_mini_cluster.cc:1428] Setting key 5e15e08ab489c5f8adda865421d4e854
I20260928 16:39:32.169337 30854 heartbeater.cc:347] Connected to a master server at 127.25.71.126:46161
I20260928 16:39:32.169716 30854 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:32.170347 30854 heartbeater.cc:511] Master 127.25.71.126:46161 requested a full tablet report, sending...
I20260928 16:39:32.171967 30533 ts_manager.cc:194] Registered new tserver with Master: 4ec8b760dafa4f3893190714d50e7baa (127.25.71.66:35403)
I20260928 16:39:32.174211 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:46161
--builtin_ntp_servers=127.25.71.84:42123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 16:39:32.313481 30857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:32.313862 30857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:32.313917 30857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:32.318511 30857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:32.318619 30857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:32.318740 30857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:39:32.324400 30857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:42123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:46161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:32.325956 30857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:32.327843 30857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:32.340029 30865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:32.340050 30862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:32.340040 30863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:32.340329 30857 server_base.cc:1034] running on GCE node
I20260928 16:39:32.340952 30857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:32.341737 30857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:32.342960 30857 hybrid_clock.cc:648] HybridClock initialized: now 1790613572342931 us; error 61 us; skew 500 ppm
I20260928 16:39:32.345432 30857 webserver.cc:460] Webserver started at http://127.25.71.67:38197/ using document root <none> and password file <none>
I20260928 16:39:32.346139 30857 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:32.346217 30857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:32.346437 30857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:32.348893 30857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "b8197bd5061a4ff2a22914c2992051a5"
format_stamp: "Formatted at 2026-09-28 16:39:32 on dist-test-slave-nqgh"
server_key: "de668b85ab4dadefadab75f83163cce9"
server_key_iv: "69185c81f5cc09099fa9986817bcd8c4"
server_key_version: "encryptionkey@0"
I20260928 16:39:32.349502 30857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "b8197bd5061a4ff2a22914c2992051a5"
format_stamp: "Formatted at 2026-09-28 16:39:32 on dist-test-slave-nqgh"
server_key: "de668b85ab4dadefadab75f83163cce9"
server_key_iv: "69185c81f5cc09099fa9986817bcd8c4"
server_key_version: "encryptionkey@0"
I20260928 16:39:32.354207 30857 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:39:32.357822 30871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:32.360014 30857 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:32.360201 30857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "b8197bd5061a4ff2a22914c2992051a5"
format_stamp: "Formatted at 2026-09-28 16:39:32 on dist-test-slave-nqgh"
server_key: "de668b85ab4dadefadab75f83163cce9"
server_key_iv: "69185c81f5cc09099fa9986817bcd8c4"
server_key_version: "encryptionkey@0"
I20260928 16:39:32.360380 30857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:32.365768 30857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:32.366591 30857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:32.366858 30857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:32.367480 30857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:32.368559 30857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:32.368611 30857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:32.368669 30857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:32.368697 30857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:32.378892 30857 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:38487
I20260928 16:39:32.378943 30985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:38487 every 8 connection(s)
I20260928 16:39:32.380007 30857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:39:32.383396 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 30857
I20260928 16:39:32.383520 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:39:32.383834 25885 external_mini_cluster.cc:1428] Setting key f44ca1af816787c587815fd21b49e6c3
I20260928 16:39:32.388607 30572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:32.382708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:46415 (local address 127.25.71.126:46161)
0928 16:39:32.382922 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:32.382926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:32.384784 (+ 1858us) server_negotiation.cc:407] Connection header received
0928 16:39:32.386523 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:32.386527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:32.386593 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:32.386735 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:32.388074 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:32.388078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:32.388081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:39:32.388096 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:32.388129 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:39:32.388138 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:32.388237 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:32.388390 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 16:39:32.388440 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:39:32.389576 30988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:32.383299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:46161 (local address 127.25.71.67:46415)
0928 16:39:32.384589 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:32.384670 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:32.386221 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:32.386933 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:32.386941 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:32.387053 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:32.387691 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:39:32.387693 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:32.387742 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:39:32.387852 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:32.388231 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:32.388238 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:32.388242 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 16:39:32.388433 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 16:39:32.388666 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":213,"threads_started":1}
I20260928 16:39:32.391187 30986 heartbeater.cc:347] Connected to a master server at 127.25.71.126:46161
I20260928 16:39:32.391618 30986 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:32.392228 30986 heartbeater.cc:511] Master 127.25.71.126:46161 requested a full tablet report, sending...
I20260928 16:39:32.393780 30533 ts_manager.cc:194] Registered new tserver with Master: b8197bd5061a4ff2a22914c2992051a5 (127.25.71.67:38487)
I20260928 16:39:32.399454 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:39:32.403743 30572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:32.401696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60524 (local address 127.25.71.126:46161)
0928 16:39:32.401934 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:32.401940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:32.402199 (+ 259us) server_negotiation.cc:407] Connection header received
0928 16:39:32.402291 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:32.402294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:32.402356 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:32.402459 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:32.403223 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:32.403226 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:32.403229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:39:32.403241 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:32.403266 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:39:32.403275 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:32.403373 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:32.403522 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 16:39:32.403567 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 16:39:32.407783 30533 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:39:32.411924 30533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:39:32.427989 31002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:32.424832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:35403 (local address 127.0.0.1:46192)
0928 16:39:32.425422 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:32.425493 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:32.425647 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:32.426450 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:32.426454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:32.426470 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:32.426863 (+ 393us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:39:32.426864 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:32.426873 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:39:32.426964 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:32.427465 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:32.427475 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:32.427481 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:39:32.427614 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 16:39:32.427734 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":122,"threads_started":1}
I20260928 16:39:32.428028 30999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:32.423978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34162 (local address 127.25.71.67:38487)
0928 16:39:32.424529 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:32.424540 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:32.424808 (+ 268us) server_negotiation.cc:407] Connection header received
0928 16:39:32.424895 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:32.424901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:32.425199 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:32.425360 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:32.427245 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:32.427253 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:32.427272 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:39:32.427287 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:32.427344 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:39:32.427381 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:32.427495 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:32.427770 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 16:39:32.427864 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":216,"threads_started":1}
I20260928 16:39:32.428015 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:32.423863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:38487 (local address 127.0.0.1:34162)
0928 16:39:32.424388 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:32.424519 (+ 131us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:32.424676 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:32.425798 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:32.425805 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:32.425909 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:32.426792 (+ 883us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:39:32.426794 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:32.426820 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:39:32.427095 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:32.427533 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:32.427536 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:32.427539 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 16:39:32.427614 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 16:39:32.427833 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":118,"thread_start_us":84,"threads_started":1}
I20260928 16:39:32.428790 31001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:32.424838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57912 (local address 127.25.71.65:35755)
0928 16:39:32.425908 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:32.425916 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:32.425949 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:39:32.426015 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:32.426022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:32.426265 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:32.426452 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:32.427381 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:32.427387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:32.427396 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:39:32.427408 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:32.427449 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:39:32.427478 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:32.427574 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:32.428570 (+ 996us) server_negotiation.cc:299] Negotiation successful
0928 16:39:32.428654 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":954,"thread_start_us":115,"threads_started":1}
I20260928 16:39:32.428959 31003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:32.425314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46192 (local address 127.25.71.66:35403)
0928 16:39:32.426085 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:32.426096 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:32.426115 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:39:32.426164 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:32.426168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:32.426293 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:32.426452 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:32.427255 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:32.427261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:32.427272 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:39:32.427280 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:32.427326 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:39:32.427354 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:32.427496 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:32.427649 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 16:39:32.427749 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":221,"threads_started":1}
I20260928 16:39:32.429076 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:32.424007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:35755 (local address 127.0.0.1:57912)
0928 16:39:32.424927 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:32.424940 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:32.425021 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:32.426572 (+ 1551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:32.426575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:32.426589 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:32.427204 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:39:32.427206 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:32.427216 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:39:32.427237 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:32.427637 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:32.427641 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:32.427645 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 16:39:32.427732 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 16:39:32.427785 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"thread_start_us":177,"threads_started":1}
I20260928 16:39:32.432060 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff2713a78af4fc994f4c2fe9ac181e4 (DEFAULT_TABLE table=test-table [id=223c402334cf4f588be723f65100feb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:32.432415 30655 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff2713a78af4fc994f4c2fe9ac181e4 (DEFAULT_TABLE table=test-table [id=223c402334cf4f588be723f65100feb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:32.432415 30787 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff2713a78af4fc994f4c2fe9ac181e4 (DEFAULT_TABLE table=test-table [id=223c402334cf4f588be723f65100feb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:32.433672 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff2713a78af4fc994f4c2fe9ac181e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:32.433753 30655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff2713a78af4fc994f4c2fe9ac181e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:32.433753 30787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff2713a78af4fc994f4c2fe9ac181e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:32.444094 31005 tablet_bootstrap.cc:492] T 2ff2713a78af4fc994f4c2fe9ac181e4 P c66bbe8691004354b79c4f3e62b27af4: Bootstrap starting.
I20260928 16:39:32.444110 31004 tablet_bootstrap.cc:492] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5: Bootstrap starting.
I20260928 16:39:32.444639 31006 tablet_bootstrap.cc:492] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa: Bootstrap starting.
I20260928 16:39:32.447031 31004 tablet_bootstrap.cc:654] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:32.447237 31006 tablet_bootstrap.cc:654] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:32.447237 31005 tablet_bootstrap.cc:654] T 2ff2713a78af4fc994f4c2fe9ac181e4 P c66bbe8691004354b79c4f3e62b27af4: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:32.448050 31004 log.cc:824] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:32.448266 31005 log.cc:824] T 2ff2713a78af4fc994f4c2fe9ac181e4 P c66bbe8691004354b79c4f3e62b27af4: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:32.448266 31006 log.cc:824] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:32.450832 31006 tablet_bootstrap.cc:492] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa: No bootstrap required, opened a new log
I20260928 16:39:32.450832 31005 tablet_bootstrap.cc:492] T 2ff2713a78af4fc994f4c2fe9ac181e4 P c66bbe8691004354b79c4f3e62b27af4: No bootstrap required, opened a new log
I20260928 16:39:32.450976 31004 tablet_bootstrap.cc:492] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5: No bootstrap required, opened a new log
I20260928 16:39:32.451076 31005 ts_tablet_manager.cc:1397] T 2ff2713a78af4fc994f4c2fe9ac181e4 P c66bbe8691004354b79c4f3e62b27af4: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:39:32.451076 31006 ts_tablet_manager.cc:1397] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:39:32.451232 31004 ts_tablet_manager.cc:1397] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:39:32.455307 31004 raft_consensus.cc:348] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8197bd5061a4ff2a22914c2992051a5" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38487 } } peers { permanent_uuid: "c66bbe8691004354b79c4f3e62b27af4" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35755 } } peers { permanent_uuid: "4ec8b760dafa4f3893190714d50e7baa" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35403 } }
I20260928 16:39:32.455641 31004 raft_consensus.cc:374] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:32.455708 31004 raft_consensus.cc:729] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8197bd5061a4ff2a22914c2992051a5, State: Initialized, Role: FOLLOWER
I20260928 16:39:32.456259 31006 raft_consensus.cc:348] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8197bd5061a4ff2a22914c2992051a5" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38487 } } peers { permanent_uuid: "c66bbe8691004354b79c4f3e62b27af4" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35755 } } peers { permanent_uuid: "4ec8b760dafa4f3893190714d50e7baa" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35403 } }
I20260928 16:39:32.456223 31005 raft_consensus.cc:348] T 2ff2713a78af4fc994f4c2fe9ac181e4 P c66bbe8691004354b79c4f3e62b27af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8197bd5061a4ff2a22914c2992051a5" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38487 } } peers { permanent_uuid: "c66bbe8691004354b79c4f3e62b27af4" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35755 } } peers { permanent_uuid: "4ec8b760dafa4f3893190714d50e7baa" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35403 } }
I20260928 16:39:32.456595 31006 raft_consensus.cc:374] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:32.456610 31005 raft_consensus.cc:374] T 2ff2713a78af4fc994f4c2fe9ac181e4 P c66bbe8691004354b79c4f3e62b27af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:32.456650 31006 raft_consensus.cc:729] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec8b760dafa4f3893190714d50e7baa, State: Initialized, Role: FOLLOWER
I20260928 16:39:32.456326 31004 consensus_queue.cc:260] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8197bd5061a4ff2a22914c2992051a5" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38487 } } peers { permanent_uuid: "c66bbe8691004354b79c4f3e62b27af4" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35755 } } peers { permanent_uuid: "4ec8b760dafa4f3893190714d50e7baa" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35403 } }
I20260928 16:39:32.456697 31005 raft_consensus.cc:729] T 2ff2713a78af4fc994f4c2fe9ac181e4 P c66bbe8691004354b79c4f3e62b27af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c66bbe8691004354b79c4f3e62b27af4, State: Initialized, Role: FOLLOWER
I20260928 16:39:32.457247 31005 consensus_queue.cc:260] T 2ff2713a78af4fc994f4c2fe9ac181e4 P c66bbe8691004354b79c4f3e62b27af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8197bd5061a4ff2a22914c2992051a5" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38487 } } peers { permanent_uuid: "c66bbe8691004354b79c4f3e62b27af4" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35755 } } peers { permanent_uuid: "4ec8b760dafa4f3893190714d50e7baa" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35403 } }
I20260928 16:39:32.457247 31006 consensus_queue.cc:260] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8197bd5061a4ff2a22914c2992051a5" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38487 } } peers { permanent_uuid: "c66bbe8691004354b79c4f3e62b27af4" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35755 } } peers { permanent_uuid: "4ec8b760dafa4f3893190714d50e7baa" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35403 } }
I20260928 16:39:32.458017 31004 ts_tablet_manager.cc:1428] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:39:32.458370 30721 heartbeater.cc:503] Master 127.25.71.126:46161 was elected leader, sending a full tablet report...
I20260928 16:39:32.458438 30986 heartbeater.cc:503] Master 127.25.71.126:46161 was elected leader, sending a full tablet report...
I20260928 16:39:32.458565 31005 ts_tablet_manager.cc:1428] T 2ff2713a78af4fc994f4c2fe9ac181e4 P c66bbe8691004354b79c4f3e62b27af4: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 16:39:32.459050 30854 heartbeater.cc:503] Master 127.25.71.126:46161 was elected leader, sending a full tablet report...
I20260928 16:39:32.459481 31006 ts_tablet_manager.cc:1428] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
W20260928 16:39:32.631099 30987 tablet.cc:2369] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:39:32.658512 30855 tablet.cc:2369] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:39:32.681231 30722 tablet.cc:2369] T 2ff2713a78af4fc994f4c2fe9ac181e4 P c66bbe8691004354b79c4f3e62b27af4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:39:32.693380 31011 raft_consensus.cc:482] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:39:32.693719 31011 raft_consensus.cc:504] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8197bd5061a4ff2a22914c2992051a5" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38487 } } peers { permanent_uuid: "c66bbe8691004354b79c4f3e62b27af4" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35755 } } peers { permanent_uuid: "4ec8b760dafa4f3893190714d50e7baa" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35403 } }
I20260928 16:39:32.695384 31011 leader_election.cc:288] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8197bd5061a4ff2a22914c2992051a5 (127.25.71.67:38487), c66bbe8691004354b79c4f3e62b27af4 (127.25.71.65:35755)
I20260928 16:39:32.698181 30999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:32.695896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:44517 (local address 127.25.71.67:38487)
0928 16:39:32.696085 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:32.696092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:32.696376 (+ 284us) server_negotiation.cc:407] Connection header received
0928 16:39:32.696561 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:32.696564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:32.696611 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:32.696707 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:32.697630 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:32.697633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:32.697636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:39:32.697648 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:32.697672 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:39:32.697682 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:32.697779 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:32.697980 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 16:39:32.698035 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 16:39:32.698181 31001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:32.695994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:57321 (local address 127.25.71.65:35755)
0928 16:39:32.696142 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:32.696146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:32.696396 (+ 250us) server_negotiation.cc:407] Connection header received
0928 16:39:32.696560 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:32.696563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:32.696611 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:32.696706 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:32.697630 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:32.697633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:32.697636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:39:32.697648 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:32.697672 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:39:32.697682 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:32.697779 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:32.697979 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 16:39:32.698031 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 16:39:32.698271 31014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:32.695765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:38487 (local address 127.25.71.66:44517)
0928 16:39:32.696238 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:32.696256 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:32.696414 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:32.696806 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:32.696810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:32.696831 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:32.697410 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:39:32.697413 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:32.697429 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:39:32.697471 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:32.697803 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:32.697807 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:32.697811 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 16:39:32.697924 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 16:39:32.697997 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":21,"thread_start_us":153,"threads_started":1}
I20260928 16:39:32.698274 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:32.695859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:35755 (local address 127.25.71.66:57321)
0928 16:39:32.696300 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:32.696314 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:32.696414 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:32.696806 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:32.696810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:32.696830 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:32.697457 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:39:32.697459 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:32.697467 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:39:32.697484 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:32.697803 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:32.697807 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:32.697811 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 16:39:32.697924 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 16:39:32.697983 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"spinlock_wait_cycles":13696,"thread_start_us":121,"threads_started":1}
I20260928 16:39:32.699229 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff2713a78af4fc994f4c2fe9ac181e4" candidate_uuid: "4ec8b760dafa4f3893190714d50e7baa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8197bd5061a4ff2a22914c2992051a5" is_pre_election: true
I20260928 16:39:32.699694 30940 raft_consensus.cc:2454] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ec8b760dafa4f3893190714d50e7baa in term 0.
I20260928 16:39:32.699662 30675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff2713a78af4fc994f4c2fe9ac181e4" candidate_uuid: "4ec8b760dafa4f3893190714d50e7baa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c66bbe8691004354b79c4f3e62b27af4" is_pre_election: true
I20260928 16:39:32.700062 30675 raft_consensus.cc:2454] T 2ff2713a78af4fc994f4c2fe9ac181e4 P c66bbe8691004354b79c4f3e62b27af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ec8b760dafa4f3893190714d50e7baa in term 0.
I20260928 16:39:32.700314 30739 leader_election.cc:302] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ec8b760dafa4f3893190714d50e7baa, b8197bd5061a4ff2a22914c2992051a5; no voters:
I20260928 16:39:32.700645 31011 raft_consensus.cc:2781] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:39:32.700744 31011 raft_consensus.cc:482] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:39:32.700788 31011 raft_consensus.cc:3037] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:32.702100 31011 raft_consensus.cc:504] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8197bd5061a4ff2a22914c2992051a5" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38487 } } peers { permanent_uuid: "c66bbe8691004354b79c4f3e62b27af4" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35755 } } peers { permanent_uuid: "4ec8b760dafa4f3893190714d50e7baa" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35403 } }
I20260928 16:39:32.702550 31011 leader_election.cc:288] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa [CANDIDATE]: Term 1 election: Requested vote from peers b8197bd5061a4ff2a22914c2992051a5 (127.25.71.67:38487), c66bbe8691004354b79c4f3e62b27af4 (127.25.71.65:35755)
I20260928 16:39:32.703145 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff2713a78af4fc994f4c2fe9ac181e4" candidate_uuid: "4ec8b760dafa4f3893190714d50e7baa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8197bd5061a4ff2a22914c2992051a5"
I20260928 16:39:32.703161 30675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff2713a78af4fc994f4c2fe9ac181e4" candidate_uuid: "4ec8b760dafa4f3893190714d50e7baa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c66bbe8691004354b79c4f3e62b27af4"
I20260928 16:39:32.703361 30940 raft_consensus.cc:3037] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:32.703361 30675 raft_consensus.cc:3037] T 2ff2713a78af4fc994f4c2fe9ac181e4 P c66bbe8691004354b79c4f3e62b27af4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:32.704756 30675 raft_consensus.cc:2454] T 2ff2713a78af4fc994f4c2fe9ac181e4 P c66bbe8691004354b79c4f3e62b27af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ec8b760dafa4f3893190714d50e7baa in term 1.
I20260928 16:39:32.704756 30940 raft_consensus.cc:2454] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ec8b760dafa4f3893190714d50e7baa in term 1.
I20260928 16:39:32.705155 30739 leader_election.cc:302] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ec8b760dafa4f3893190714d50e7baa, b8197bd5061a4ff2a22914c2992051a5; no voters:
I20260928 16:39:32.705473 31011 raft_consensus.cc:2781] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:32.705818 31011 raft_consensus.cc:686] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa [term 1 LEADER]: Becoming Leader. State: Replica: 4ec8b760dafa4f3893190714d50e7baa, State: Running, Role: LEADER
I20260928 16:39:32.706218 31011 consensus_queue.cc:237] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8197bd5061a4ff2a22914c2992051a5" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38487 } } peers { permanent_uuid: "c66bbe8691004354b79c4f3e62b27af4" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35755 } } peers { permanent_uuid: "4ec8b760dafa4f3893190714d50e7baa" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35403 } }
I20260928 16:39:32.710316 30531 catalog_manager.cc:5495] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ec8b760dafa4f3893190714d50e7baa (127.25.71.66). New cstate: current_term: 1 leader_uuid: "4ec8b760dafa4f3893190714d50e7baa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8197bd5061a4ff2a22914c2992051a5" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c66bbe8691004354b79c4f3e62b27af4" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ec8b760dafa4f3893190714d50e7baa" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 35403 } health_report { overall_health: HEALTHY } } }
I20260928 16:39:32.733698 31003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:32.731559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46198 (local address 127.25.71.66:35403)
0928 16:39:32.731894 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:32.731899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:32.731919 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:39:32.732026 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:32.732029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:32.732105 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:32.732206 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:32.733173 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:32.733176 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:32.733179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:39:32.733190 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:32.733212 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:39:32.733221 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:32.733322 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:32.733489 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 16:39:32.733539 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260928 16:39:32.740343 30940 raft_consensus.cc:1261] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5 [term 1 FOLLOWER]: Refusing update from remote peer 4ec8b760dafa4f3893190714d50e7baa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:32.740517 30675 raft_consensus.cc:1261] T 2ff2713a78af4fc994f4c2fe9ac181e4 P c66bbe8691004354b79c4f3e62b27af4 [term 1 FOLLOWER]: Refusing update from remote peer 4ec8b760dafa4f3893190714d50e7baa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:32.741145 31015 consensus_queue.cc:1035] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8197bd5061a4ff2a22914c2992051a5" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:32.741361 31011 consensus_queue.cc:1035] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c66bbe8691004354b79c4f3e62b27af4" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:32.752235 31018 mvcc.cc:205] Tried to move back new op lower bound from 7334353193936494592 to 7334353193808363520. Current Snapshot: MvccSnapshot[applied={T|T < 7334353193936494592}]
I20260928 16:39:32.753748 31020 mvcc.cc:205] Tried to move back new op lower bound from 7334353193936494592 to 7334353193808363520. Current Snapshot: MvccSnapshot[applied={T|T < 7334353193936494592}]
I20260928 16:39:32.755147 31019 mvcc.cc:205] Tried to move back new op lower bound from 7334353193936494592 to 7334353193808363520. Current Snapshot: MvccSnapshot[applied={T|T < 7334353193936494592}]
I20260928 16:39:32.764029 30533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:32.764326 30533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:32.767678 30533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9da726416281415e80afe3363f105f9b: Sending DeleteTablet for 3 replicas of tablet 2ff2713a78af4fc994f4c2fe9ac181e4
I20260928 16:39:32.768746 30655 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff2713a78af4fc994f4c2fe9ac181e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:32 UTC) from {username='slave'} at 127.0.0.1:57912
I20260928 16:39:32.768746 30787 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff2713a78af4fc994f4c2fe9ac181e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:32 UTC) from {username='slave'} at 127.0.0.1:46192
I20260928 16:39:32.769440 31028 tablet_replica.cc:330] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa: stopping tablet replica
I20260928 16:39:32.769526 30920 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff2713a78af4fc994f4c2fe9ac181e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:32 UTC) from {username='slave'} at 127.0.0.1:34162
I20260928 16:39:32.769886 31029 tablet_replica.cc:330] T 2ff2713a78af4fc994f4c2fe9ac181e4 P c66bbe8691004354b79c4f3e62b27af4: stopping tablet replica
I20260928 16:39:32.769968 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 30593
I20260928 16:39:32.770771 31028 raft_consensus.cc:2229] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:39:32.771380 31028 raft_consensus.cc:2258] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:32.772325 31030 tablet_replica.cc:330] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5: stopping tablet replica
I20260928 16:39:32.772739 31030 raft_consensus.cc:2229] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:32.773142 31030 raft_consensus.cc:2258] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:32.775022 31028 ts_tablet_manager.cc:1905] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:32.775054 31030 ts_tablet_manager.cc:1905] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 16:39:32.778546 30519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.71.65:35755 (error 108)
I20260928 16:39:32.778721 31030 ts_tablet_manager.cc:1918] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:39:32.778859 31030 log.cc:1196] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/2ff2713a78af4fc994f4c2fe9ac181e4
I20260928 16:39:32.779002 31028 ts_tablet_manager.cc:1918] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:39:32.779208 31028 log.cc:1196] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/2ff2713a78af4fc994f4c2fe9ac181e4
I20260928 16:39:32.779209 31030 ts_tablet_manager.cc:1939] T 2ff2713a78af4fc994f4c2fe9ac181e4 P b8197bd5061a4ff2a22914c2992051a5: Deleting consensus metadata
I20260928 16:39:32.779601 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 30724
I20260928 16:39:32.779655 31028 ts_tablet_manager.cc:1939] T 2ff2713a78af4fc994f4c2fe9ac181e4 P 4ec8b760dafa4f3893190714d50e7baa: Deleting consensus metadata
I20260928 16:39:32.784547 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:32.784034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:35755 (local address 127.0.0.1:57914)
0928 16:39:32.784290 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:32.784438 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.65:35755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260928 16:39:32.784834 30519 catalog_manager.cc:4584] TS c66bbe8691004354b79c4f3e62b27af4 (127.25.71.65:35755): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ff2713a78af4fc994f4c2fe9ac181e4: Network error: Client connection negotiation failed: client connection to 127.25.71.65:35755: connect: Connection refused (error 111)
I20260928 16:39:32.785107 30517 catalog_manager.cc:4841] TS b8197bd5061a4ff2a22914c2992051a5 (127.25.71.67:38487): tablet 2ff2713a78af4fc994f4c2fe9ac181e4 (table test-table [id=223c402334cf4f588be723f65100feb7]) successfully deleted
I20260928 16:39:32.790299 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 30857
I20260928 16:39:32.792599 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:32.790548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:35403 (local address 127.0.0.1:46200)
0928 16:39:32.792402 (+ 1854us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:32.792475 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.66:35403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1717}
I20260928 16:39:32.796573 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 30502
2026-09-28T16:39:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1405 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 16:39:32.820047 25885 mini_oidc.cc:152] Starting JWKS server
I20260928 16:39:32.822218 25885 webserver.cc:460] Webserver started at https://127.0.0.1:38105/ using document root <none> and password file <none>
I20260928 16:39:32.822417 25885 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 16:39:32.822902 25885 webserver.cc:460] Webserver started at http://127.0.0.1:33631/ using document root <none> and password file <none>
2026-09-28T16:39:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:39:32Z Disabled control of system clock
I20260928 16:39:32.835314 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:33003
--webserver_interface=127.25.71.126
--webserver_port=0
--builtin_ntp_servers=127.25.71.84:44861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:33003
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38105/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613545667819-25885-0/testchainca.cert with env {}
W20260928 16:39:32.977388 31037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:32.977834 31037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:32.977885 31037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:32.982944 31037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:32.983035 31037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:32.983050 31037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:32.983068 31037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:32.983083 31037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 16:39:32.983121 31037 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 16:39:32.983141 31037 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38105/jwks.json
I20260928 16:39:32.990880 31037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:44861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:33003
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:33003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38105/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613545667819-25885-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:32.992465 31037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:32.993836 31037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:33.005486 31043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:33.005607 31037 server_base.cc:1034] running on GCE node
W20260928 16:39:33.005468 31045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:33.005522 31042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:33.006811 31037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:33.008577 31037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:33.009831 31037 hybrid_clock.cc:648] HybridClock initialized: now 1790613573009804 us; error 53 us; skew 500 ppm
I20260928 16:39:33.012671 31037 webserver.cc:460] Webserver started at http://127.25.71.126:40907/ using document root <none> and password file <none>
I20260928 16:39:33.013679 31037 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:33.013816 31037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:33.014083 31037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:33.016973 31037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "eea979843c1d4d1288add56e1ca4ac79"
format_stamp: "Formatted at 2026-09-28 16:39:33 on dist-test-slave-nqgh"
server_key: "3bad470597cbe99e17000d98516d688a"
server_key_iv: "8107b7352fd1da068b0a2d5ada3321a4"
server_key_version: "encryptionkey@0"
I20260928 16:39:33.017675 31037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "eea979843c1d4d1288add56e1ca4ac79"
format_stamp: "Formatted at 2026-09-28 16:39:33 on dist-test-slave-nqgh"
server_key: "3bad470597cbe99e17000d98516d688a"
server_key_iv: "8107b7352fd1da068b0a2d5ada3321a4"
server_key_version: "encryptionkey@0"
I20260928 16:39:33.023201 31037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 16:39:33.027300 31052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:33.029217 31037 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:33.029422 31037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "eea979843c1d4d1288add56e1ca4ac79"
format_stamp: "Formatted at 2026-09-28 16:39:33 on dist-test-slave-nqgh"
server_key: "3bad470597cbe99e17000d98516d688a"
server_key_iv: "8107b7352fd1da068b0a2d5ada3321a4"
server_key_version: "encryptionkey@0"
I20260928 16:39:33.029584 31037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:33.047727 31037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:33.048743 31037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:33.049115 31037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:33.058660 31037 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:33003
I20260928 16:39:33.058739 31104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:33003 every 8 connection(s)
I20260928 16:39:33.060006 31037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:39:33.064065 31105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:33.065249 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 31037
I20260928 16:39:33.065395 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:39:33.065747 25885 external_mini_cluster.cc:1428] Setting key 11876d2fbde1c3b43d2a27b27b4742a0
I20260928 16:39:33.072290 31105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79: Bootstrap starting.
I20260928 16:39:33.075131 31105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:33.076100 31105 log.cc:824] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:33.079368 31105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79: No bootstrap required, opened a new log
I20260928 16:39:33.079376 31108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:33.067803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.25.71.126:33003)
0928 16:39:33.068771 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:33.068796 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:33.068853 (+ 57us) server_negotiation.cc:407] Connection header received
0928 16:39:33.069990 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:33.070021 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:33.070667 (+ 646us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:33.071129 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:33.071672 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:33.073126 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:33.074421 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:33.075190 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:33.075388 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:33.076627 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:33.076633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:33.076651 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:39:33.077319 (+ 668us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:33.077415 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:39:33.077449 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:33.077584 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:33.077853 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 16:39:33.078213 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":194,"threads_started":1}
I20260928 16:39:33.083622 31105 raft_consensus.cc:348] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea979843c1d4d1288add56e1ca4ac79" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33003 } }
I20260928 16:39:33.084003 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:39:33.084111 31105 raft_consensus.cc:374] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:33.084197 31105 raft_consensus.cc:729] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea979843c1d4d1288add56e1ca4ac79, State: Initialized, Role: FOLLOWER
I20260928 16:39:33.084872 31105 consensus_queue.cc:260] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea979843c1d4d1288add56e1ca4ac79" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33003 } }
I20260928 16:39:33.085232 31105 raft_consensus.cc:388] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:39:33.085335 31105 raft_consensus.cc:482] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:39:33.085615 31105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:33.086942 31105 raft_consensus.cc:504] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea979843c1d4d1288add56e1ca4ac79" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33003 } }
I20260928 16:39:33.087423 31105 leader_election.cc:302] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eea979843c1d4d1288add56e1ca4ac79; no voters:
I20260928 16:39:33.087793 31105 leader_election.cc:288] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:39:33.088047 31110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:33.088377 31110 raft_consensus.cc:686] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79 [term 1 LEADER]: Becoming Leader. State: Replica: eea979843c1d4d1288add56e1ca4ac79, State: Running, Role: LEADER
I20260928 16:39:33.089298 31110 consensus_queue.cc:237] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea979843c1d4d1288add56e1ca4ac79" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33003 } }
I20260928 16:39:33.089663 31105 sys_catalog.cc:564] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:33.091897 31112 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eea979843c1d4d1288add56e1ca4ac79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea979843c1d4d1288add56e1ca4ac79" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33003 } } }
I20260928 16:39:33.091918 31111 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eea979843c1d4d1288add56e1ca4ac79. Latest consensus state: current_term: 1 leader_uuid: "eea979843c1d4d1288add56e1ca4ac79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea979843c1d4d1288add56e1ca4ac79" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33003 } } }
I20260928 16:39:33.092154 31112 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:33.092152 31111 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79 [sys.catalog]: This master's current role is: LEADER
W20260928 16:39:33.095577 31124 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:39:33.095702 31124 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:39:33.095777 31125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:39:33.096694 31125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:39:33.104799 31125 catalog_manager.cc:1347] Generated new cluster ID: c5c3b0e81d7c4955a2fccf103398656f
I20260928 16:39:33.104923 31125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:39:33.118968 31125 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:39:33.120663 31125 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:39:33.130666 31125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eea979843c1d4d1288add56e1ca4ac79: Generated new TSK 0
I20260928 16:39:33.132066 31125 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 16:39:33.173141 31135 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 16:39:33.173411 31135 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 16:39:33.174085 31108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:33.154011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49804 (local address 127.25.71.126:33003)
0928 16:39:33.154163 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:33.154167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:33.154389 (+ 222us) server_negotiation.cc:407] Connection header received
0928 16:39:33.154707 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:33.154711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:33.154791 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:33.154962 (+ 171us) server_negotiation.cc:226] Negotiated authn=JWT
0928 16:39:33.155356 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:33.156630 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:33.157813 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:33.158210 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:33.158348 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:33.159088 (+ 740us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 16:39:33.173324 (+ 14236us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 16:39:33.173895 (+ 571us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49804: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 16:39:33.174218 31108 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49804: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 16:39:33.175786 31134 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:33.153833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33003 (local address 127.0.0.1:49804)
0928 16:39:33.154260 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:33.154277 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:33.154450 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:33.155032 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:33.155035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:33.155051 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0928 16:39:33.155205 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:33.155212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:33.156827 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:33.156832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:33.157633 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:33.157645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:33.158484 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:33.158492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:33.158674 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:33.158769 (+ 95us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 16:39:33.173854 (+ 15085us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 16:39:33.174241 (+ 387us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.71.126:33003: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":144,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.25.71.126:33003: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 16:39:33.177012 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 31037
2026-09-28T16:39:33Z chronyd exiting
I20260928 16:39:33.217272 25885 test_util.cc:175] -----------------------------------------------
I20260928 16:39:33.217399 25885 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613545667819-25885-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (399 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 16:39:33.218711 25885 mini_oidc.cc:152] Starting JWKS server
I20260928 16:39:33.219751 25885 webserver.cc:460] Webserver started at https://127.0.0.1:43787/ using document root <none> and password file <none>
I20260928 16:39:33.219851 25885 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 16:39:33.220124 25885 webserver.cc:460] Webserver started at http://127.0.0.1:46043/ using document root <none> and password file <none>
2026-09-28T16:39:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:39:33Z Disabled control of system clock
I20260928 16:39:33.232420 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:37515
--webserver_interface=127.25.71.126
--webserver_port=0
--builtin_ntp_servers=127.25.71.84:39959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:37515
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43787/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613545667819-25885-0/testchainca.cert with env {}
W20260928 16:39:33.377408 31141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:33.377841 31141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:33.377893 31141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:33.382717 31141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:33.382843 31141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:33.382858 31141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:33.382877 31141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:33.382892 31141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 16:39:33.382910 31141 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 16:39:33.382922 31141 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43787/jwks.json
I20260928 16:39:33.390465 31141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:39959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:37515
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:37515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43787/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613545667819-25885-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:33.392349 31141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:33.394145 31141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:33.406101 31149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:33.406080 31141 server_base.cc:1034] running on GCE node
W20260928 16:39:33.406085 31146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:33.406047 31147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:33.407225 31141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:33.408618 31141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:33.409894 31141 hybrid_clock.cc:648] HybridClock initialized: now 1790613573409871 us; error 52 us; skew 500 ppm
I20260928 16:39:33.412236 31141 webserver.cc:460] Webserver started at http://127.25.71.126:38681/ using document root <none> and password file <none>
I20260928 16:39:33.412950 31141 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:33.413026 31141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:33.413242 31141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:33.416047 31141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "b38cf5a7542a4e2a8e36949056c0a326"
format_stamp: "Formatted at 2026-09-28 16:39:33 on dist-test-slave-nqgh"
server_key: "ee1830c0f62720233801e1df74879170"
server_key_iv: "834a72eb1e13fe50afdb0ad271d62e55"
server_key_version: "encryptionkey@0"
I20260928 16:39:33.416677 31141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "b38cf5a7542a4e2a8e36949056c0a326"
format_stamp: "Formatted at 2026-09-28 16:39:33 on dist-test-slave-nqgh"
server_key: "ee1830c0f62720233801e1df74879170"
server_key_iv: "834a72eb1e13fe50afdb0ad271d62e55"
server_key_version: "encryptionkey@0"
I20260928 16:39:33.421520 31141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:39:33.425036 31155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:33.427171 31141 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:39:33.427367 31141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "b38cf5a7542a4e2a8e36949056c0a326"
format_stamp: "Formatted at 2026-09-28 16:39:33 on dist-test-slave-nqgh"
server_key: "ee1830c0f62720233801e1df74879170"
server_key_iv: "834a72eb1e13fe50afdb0ad271d62e55"
server_key_version: "encryptionkey@0"
I20260928 16:39:33.427520 31141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:33.448195 31141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:33.449005 31141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:33.449227 31141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:33.457057 31141 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:37515
I20260928 16:39:33.457185 31208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:37515 every 8 connection(s)
I20260928 16:39:33.458244 31141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:39:33.462313 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 31141
I20260928 16:39:33.462488 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:39:33.462954 25885 external_mini_cluster.cc:1428] Setting key c4321aeadc0d0a09122bcbf55eadbb5a
I20260928 16:39:33.463789 31210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:33.473192 31210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326: Bootstrap starting.
I20260928 16:39:33.476644 31213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:33.465203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35402 (local address 127.25.71.126:37515)
0928 16:39:33.466352 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:33.466376 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:33.466429 (+ 53us) server_negotiation.cc:407] Connection header received
0928 16:39:33.467823 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:33.467850 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:33.468401 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:33.468799 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:33.469385 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:33.470770 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:33.471812 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:33.472433 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:33.472617 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:33.473642 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:33.473651 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:33.473675 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:39:33.474342 (+ 667us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:33.474454 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:39:33.474491 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:33.474694 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:33.474951 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 16:39:33.475380 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":250,"threads_started":1}
I20260928 16:39:33.476912 31210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:33.478096 31210 log.cc:824] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:33.481480 31210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326: No bootstrap required, opened a new log
I20260928 16:39:33.481395 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:39:33.486366 31210 raft_consensus.cc:348] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b38cf5a7542a4e2a8e36949056c0a326" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37515 } }
I20260928 16:39:33.486795 31210 raft_consensus.cc:374] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:33.486871 31210 raft_consensus.cc:729] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b38cf5a7542a4e2a8e36949056c0a326, State: Initialized, Role: FOLLOWER
I20260928 16:39:33.487468 31210 consensus_queue.cc:260] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b38cf5a7542a4e2a8e36949056c0a326" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37515 } }
I20260928 16:39:33.487641 31210 raft_consensus.cc:388] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:39:33.487722 31210 raft_consensus.cc:482] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:39:33.487851 31210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:33.489332 31210 raft_consensus.cc:504] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b38cf5a7542a4e2a8e36949056c0a326" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37515 } }
I20260928 16:39:33.489831 31210 leader_election.cc:302] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b38cf5a7542a4e2a8e36949056c0a326; no voters:
I20260928 16:39:33.490252 31210 leader_election.cc:288] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:39:33.490432 31215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:33.490810 31215 raft_consensus.cc:686] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326 [term 1 LEADER]: Becoming Leader. State: Replica: b38cf5a7542a4e2a8e36949056c0a326, State: Running, Role: LEADER
I20260928 16:39:33.491222 31215 consensus_queue.cc:237] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b38cf5a7542a4e2a8e36949056c0a326" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37515 } }
I20260928 16:39:33.491935 31210 sys_catalog.cc:564] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:33.493512 31216 sys_catalog.cc:455] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b38cf5a7542a4e2a8e36949056c0a326" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b38cf5a7542a4e2a8e36949056c0a326" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37515 } } }
I20260928 16:39:33.493587 31217 sys_catalog.cc:455] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b38cf5a7542a4e2a8e36949056c0a326. Latest consensus state: current_term: 1 leader_uuid: "b38cf5a7542a4e2a8e36949056c0a326" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b38cf5a7542a4e2a8e36949056c0a326" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37515 } } }
I20260928 16:39:33.493822 31216 sys_catalog.cc:458] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:33.493854 31217 sys_catalog.cc:458] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:33.494468 31224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:39:33.498224 31224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:39:33.505997 31224 catalog_manager.cc:1347] Generated new cluster ID: 3eba42ce5bf74d14b71ad657e9a8e8a2
I20260928 16:39:33.506099 31224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:39:33.513790 31224 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:39:33.515524 31224 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:39:33.530232 31224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b38cf5a7542a4e2a8e36949056c0a326: Generated new TSK 0
I20260928 16:39:33.531589 31224 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 16:39:33.563261 31240 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 16:39:33.563373 31240 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 16:39:33.564405 31213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:33.546655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35416 (local address 127.25.71.126:37515)
0928 16:39:33.546838 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:33.546844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:33.547069 (+ 225us) server_negotiation.cc:407] Connection header received
0928 16:39:33.547272 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:33.547276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:33.547358 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:33.547532 (+ 174us) server_negotiation.cc:226] Negotiated authn=JWT
0928 16:39:33.547786 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:33.548676 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:33.549671 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:33.550032 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:33.550157 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:33.550508 (+ 351us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 16:39:33.563829 (+ 13321us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43787/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 16:39:33.564214 (+ 385us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35416: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43787/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 16:39:33.564512 31213 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35416: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43787/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 16:39:33.564518 31239 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:33.546418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37515 (local address 127.0.0.1:35416)
0928 16:39:33.546908 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:33.546922 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:33.547107 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:33.547538 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:33.547540 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:33.547546 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0928 16:39:33.547675 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:33.547680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:33.548815 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:33.548818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:33.549536 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:33.549545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:33.550177 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:33.550180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:33.550253 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:33.550282 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 16:39:33.564125 (+ 13843us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43787/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 16:39:33.564341 (+ 216us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.71.126:37515: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43787/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":174,"threads_started":1}
I20260928 16:39:33.565728 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 31141
2026-09-28T16:39:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (388 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 16:39:33.606711 25885 mini_oidc.cc:152] Starting JWKS server
I20260928 16:39:33.607620 25885 webserver.cc:460] Webserver started at https://127.0.0.1:39433/ using document root <none> and password file <none>
I20260928 16:39:33.607708 25885 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 16:39:33.607925 25885 webserver.cc:460] Webserver started at http://127.0.0.1:42649/ using document root <none> and password file <none>
2026-09-28T16:39:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:39:33Z Disabled control of system clock
I20260928 16:39:33.618331 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:42499
--webserver_interface=127.25.71.126
--webserver_port=0
--builtin_ntp_servers=127.25.71.84:40541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:42499
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39433/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 16:39:33.744493 31246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:33.744905 31246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:33.744951 31246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:33.749509 31246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:33.749619 31246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:33.749639 31246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:33.749661 31246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:33.749677 31246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 16:39:33.749693 31246 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 16:39:33.749706 31246 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39433/jwks.json
I20260928 16:39:33.755836 31246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:40541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:42499
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:42499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39433/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:33.757726 31246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:33.759627 31246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:33.771942 31254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:33.771973 31252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:33.771942 31251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:33.772480 31246 server_base.cc:1034] running on GCE node
I20260928 16:39:33.773414 31246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:33.775313 31246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:33.776543 31246 hybrid_clock.cc:648] HybridClock initialized: now 1790613573776502 us; error 56 us; skew 500 ppm
I20260928 16:39:33.784760 31246 webserver.cc:460] Webserver started at http://127.25.71.126:37189/ using document root <none> and password file <none>
I20260928 16:39:33.785630 31246 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:33.785724 31246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:33.785982 31246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:33.788796 31246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "10dfb25067a74d80959d31fa0d2a4257"
format_stamp: "Formatted at 2026-09-28 16:39:33 on dist-test-slave-nqgh"
server_key: "b2434d2f672bd6938a93c6ed5c678652"
server_key_iv: "54918e6b90f7c5969c273c13d17457f8"
server_key_version: "encryptionkey@0"
I20260928 16:39:33.789448 31246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "10dfb25067a74d80959d31fa0d2a4257"
format_stamp: "Formatted at 2026-09-28 16:39:33 on dist-test-slave-nqgh"
server_key: "b2434d2f672bd6938a93c6ed5c678652"
server_key_iv: "54918e6b90f7c5969c273c13d17457f8"
server_key_version: "encryptionkey@0"
I20260928 16:39:33.794602 31246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 16:39:33.798381 31260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:33.800328 31246 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:33.800536 31246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "10dfb25067a74d80959d31fa0d2a4257"
format_stamp: "Formatted at 2026-09-28 16:39:33 on dist-test-slave-nqgh"
server_key: "b2434d2f672bd6938a93c6ed5c678652"
server_key_iv: "54918e6b90f7c5969c273c13d17457f8"
server_key_version: "encryptionkey@0"
I20260928 16:39:33.800722 31246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:33.828892 31246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:33.829928 31246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:33.830164 31246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:33.839349 31246 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:42499
I20260928 16:39:33.839447 31312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:42499 every 8 connection(s)
I20260928 16:39:33.840663 31246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:39:33.844532 31313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:33.847505 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 31246
I20260928 16:39:33.847705 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:39:33.848016 25885 external_mini_cluster.cc:1428] Setting key 986967054d01fcb9a0b9ecc7764dac78
I20260928 16:39:33.851336 31313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257: Bootstrap starting.
I20260928 16:39:33.853865 31313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:33.854663 31313 log.cc:824] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:33.857443 31313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257: No bootstrap required, opened a new log
I20260928 16:39:33.859987 31316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:33.849805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37850 (local address 127.25.71.126:42499)
0928 16:39:33.850755 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:33.850779 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:33.850838 (+ 59us) server_negotiation.cc:407] Connection header received
0928 16:39:33.851705 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:33.851730 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:33.852271 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:33.852657 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:33.853191 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:33.854368 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:33.855530 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:33.856275 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:33.856519 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:33.857610 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:33.857617 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:33.857635 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:39:33.858186 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:33.858270 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:39:33.858299 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:33.858429 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:33.858607 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 16:39:33.858940 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":151,"threads_started":1}
I20260928 16:39:33.862133 31313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10dfb25067a74d80959d31fa0d2a4257" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42499 } }
I20260928 16:39:33.862416 31313 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:33.862452 31313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10dfb25067a74d80959d31fa0d2a4257, State: Initialized, Role: FOLLOWER
I20260928 16:39:33.863037 31313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10dfb25067a74d80959d31fa0d2a4257" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42499 } }
I20260928 16:39:33.863229 31313 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:39:33.863306 31313 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:39:33.863404 31313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:33.864300 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:39:33.864780 31313 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10dfb25067a74d80959d31fa0d2a4257" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42499 } }
I20260928 16:39:33.865262 31313 leader_election.cc:302] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10dfb25067a74d80959d31fa0d2a4257; no voters:
I20260928 16:39:33.865691 31313 leader_election.cc:288] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:39:33.865913 31318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:33.866241 31318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257 [term 1 LEADER]: Becoming Leader. State: Replica: 10dfb25067a74d80959d31fa0d2a4257, State: Running, Role: LEADER
I20260928 16:39:33.866812 31318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10dfb25067a74d80959d31fa0d2a4257" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42499 } }
I20260928 16:39:33.867261 31313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:33.869087 31320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10dfb25067a74d80959d31fa0d2a4257. Latest consensus state: current_term: 1 leader_uuid: "10dfb25067a74d80959d31fa0d2a4257" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10dfb25067a74d80959d31fa0d2a4257" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42499 } } }
I20260928 16:39:33.869115 31319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10dfb25067a74d80959d31fa0d2a4257" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10dfb25067a74d80959d31fa0d2a4257" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42499 } } }
I20260928 16:39:33.869345 31320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:33.869345 31319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257 [sys.catalog]: This master's current role is: LEADER
W20260928 16:39:33.872900 31333 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:39:33.873015 31333 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:39:33.873081 31332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:39:33.873914 31332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:39:33.880117 31332 catalog_manager.cc:1347] Generated new cluster ID: d537ce635ae843debcde6b29c76736ca
I20260928 16:39:33.880223 31332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:39:33.889219 31332 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:39:33.890322 31332 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:39:33.898789 31332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10dfb25067a74d80959d31fa0d2a4257: Generated new TSK 0
I20260928 16:39:33.899585 31332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:39:33.921036 31316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:33.915187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37864 (local address 127.25.71.126:42499)
0928 16:39:33.915384 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:33.915388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:33.915580 (+ 192us) server_negotiation.cc:407] Connection header received
0928 16:39:33.915807 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:33.915809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:33.915876 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:33.915995 (+ 119us) server_negotiation.cc:226] Negotiated authn=JWT
0928 16:39:33.916375 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:33.916982 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:33.918399 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:33.918695 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:33.918813 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:33.919238 (+ 425us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 16:39:33.920632 (+ 1394us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39433/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 16:39:33.920879 (+ 247us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37864: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39433/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 16:39:33.921118 31316 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37864: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39433/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 16:39:33.924005 31342 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:33.915002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:42499 (local address 127.0.0.1:37864)
0928 16:39:33.915437 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:33.915456 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:33.915649 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:33.916018 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:33.916021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:33.916029 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 16:39:33.916192 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:33.916199 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:33.917115 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:33.917118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:33.918220 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:33.918230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:33.918844 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:33.918849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:33.918951 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:33.918991 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 16:39:33.920843 (+ 1852us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39433/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 16:39:33.923809 (+ 2966us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.71.126:42499: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39433/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":2621,"thread_start_us":159,"threads_started":1}
I20260928 16:39:33.924499 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 31246
E20260928 16:39:33.936012 31343 webserver.cc:533] Webserver: sslize failed:
E20260928 16:39:33.936152 31343 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T16:39:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (359 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T16:39:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:39:33Z Disabled control of system clock
I20260928 16:39:33.977337 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:45583
--webserver_interface=127.25.71.126
--webserver_port=0
--builtin_ntp_servers=127.25.71.84:39591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:45583
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613545667819-25885-0/testchainca.cert with env {}
W20260928 16:39:34.096882 31347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:34.097209 31347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:34.097256 31347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:34.101754 31347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:34.101845 31347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:34.101859 31347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:34.101876 31347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:34.101890 31347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 16:39:34.101905 31347 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 16:39:34.101918 31347 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 16:39:34.108042 31347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:39591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:45583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:45583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613545667819-25885-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:34.109284 31347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:34.110395 31347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:34.120365 31353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:34.120393 31352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:34.120484 31347 server_base.cc:1034] running on GCE node
W20260928 16:39:34.120388 31355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:34.121505 31347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:34.123121 31347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:34.124343 31347 hybrid_clock.cc:648] HybridClock initialized: now 1790613574124300 us; error 64 us; skew 500 ppm
I20260928 16:39:34.127264 31347 webserver.cc:460] Webserver started at http://127.25.71.126:45375/ using document root <none> and password file <none>
I20260928 16:39:34.128084 31347 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:34.128187 31347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:34.128448 31347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:34.131415 31347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "ac316d97729341d5b71c63bdadf6b865"
format_stamp: "Formatted at 2026-09-28 16:39:34 on dist-test-slave-nqgh"
server_key: "99c6afdc4a725afaf160a27da4cd3ff0"
server_key_iv: "77d2ae3cac784150a7f24195ce1de849"
server_key_version: "encryptionkey@0"
I20260928 16:39:34.132206 31347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "ac316d97729341d5b71c63bdadf6b865"
format_stamp: "Formatted at 2026-09-28 16:39:34 on dist-test-slave-nqgh"
server_key: "99c6afdc4a725afaf160a27da4cd3ff0"
server_key_iv: "77d2ae3cac784150a7f24195ce1de849"
server_key_version: "encryptionkey@0"
I20260928 16:39:34.137550 31347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:39:34.141776 31362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:34.143766 31347 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:34.144002 31347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "ac316d97729341d5b71c63bdadf6b865"
format_stamp: "Formatted at 2026-09-28 16:39:34 on dist-test-slave-nqgh"
server_key: "99c6afdc4a725afaf160a27da4cd3ff0"
server_key_iv: "77d2ae3cac784150a7f24195ce1de849"
server_key_version: "encryptionkey@0"
I20260928 16:39:34.144182 31347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:34.168877 31347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:34.169915 31347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:34.170138 31347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:34.179237 31347 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:45583
I20260928 16:39:34.179322 31414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:45583 every 8 connection(s)
I20260928 16:39:34.180685 31347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:39:34.184940 31415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:34.185210 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 31347
I20260928 16:39:34.185351 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:39:34.185724 25885 external_mini_cluster.cc:1428] Setting key b3ec85f6605870d0db4a88578ee715da
I20260928 16:39:34.192181 31415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865: Bootstrap starting.
I20260928 16:39:34.195186 31415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:34.196060 31415 log.cc:824] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:34.198222 31418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:34.187668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45498 (local address 127.25.71.126:45583)
0928 16:39:34.188547 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:34.188569 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:34.188619 (+ 50us) server_negotiation.cc:407] Connection header received
0928 16:39:34.189507 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:34.189533 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:34.190110 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:34.190520 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:34.191110 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:34.192225 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:34.193633 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:34.194462 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:34.194761 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:34.195929 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:34.195939 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:34.195959 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:39:34.196534 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:34.196621 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:39:34.196648 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:34.196801 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:34.197014 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 16:39:34.197345 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":169,"threads_started":1}
I20260928 16:39:34.199218 31415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865: No bootstrap required, opened a new log
I20260928 16:39:34.202106 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:39:34.204607 31415 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac316d97729341d5b71c63bdadf6b865" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 45583 } }
I20260928 16:39:34.204948 31415 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:34.204986 31415 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac316d97729341d5b71c63bdadf6b865, State: Initialized, Role: FOLLOWER
I20260928 16:39:34.205478 31415 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac316d97729341d5b71c63bdadf6b865" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 45583 } }
I20260928 16:39:34.205621 31415 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:39:34.205677 31415 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:39:34.205765 31415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:34.207043 31415 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac316d97729341d5b71c63bdadf6b865" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 45583 } }
I20260928 16:39:34.207469 31415 leader_election.cc:302] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac316d97729341d5b71c63bdadf6b865; no voters:
I20260928 16:39:34.207875 31415 leader_election.cc:288] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:39:34.208182 31420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:34.208503 31420 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865 [term 1 LEADER]: Becoming Leader. State: Replica: ac316d97729341d5b71c63bdadf6b865, State: Running, Role: LEADER
I20260928 16:39:34.208971 31420 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac316d97729341d5b71c63bdadf6b865" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 45583 } }
I20260928 16:39:34.209640 31415 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:34.211022 31422 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac316d97729341d5b71c63bdadf6b865. Latest consensus state: current_term: 1 leader_uuid: "ac316d97729341d5b71c63bdadf6b865" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac316d97729341d5b71c63bdadf6b865" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 45583 } } }
I20260928 16:39:34.211249 31422 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:34.211649 31421 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac316d97729341d5b71c63bdadf6b865" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac316d97729341d5b71c63bdadf6b865" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 45583 } } }
I20260928 16:39:34.211781 31421 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:34.212816 31429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:39:34.216507 31429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:39:34.224395 31429 catalog_manager.cc:1347] Generated new cluster ID: d4b34b6a612f4a29a22b84e4acb783f1
I20260928 16:39:34.224517 31429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:39:34.232682 31429 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:39:34.233678 31429 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:39:34.240581 31429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac316d97729341d5b71c63bdadf6b865: Generated new TSK 0
I20260928 16:39:34.241434 31429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:39:34.333973 31418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:34.256328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45510 (local address 127.25.71.126:45583)
0928 16:39:34.256515 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:34.256522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:34.256803 (+ 281us) server_negotiation.cc:407] Connection header received
0928 16:39:34.257078 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:34.257082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:34.257149 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:34.257287 (+ 138us) server_negotiation.cc:226] Negotiated authn=JWT
0928 16:39:34.257630 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:34.258252 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:34.259698 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:34.259976 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:34.260075 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:34.260493 (+ 418us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 16:39:34.333534 (+ 73041us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 16:39:34.333833 (+ 299us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45510: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 16:39:34.334064 31418 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45510: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 16:39:34.335055 31444 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:34.256180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:45583 (local address 127.0.0.1:45510)
0928 16:39:34.256648 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:34.256678 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:34.256870 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:34.257307 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:34.257311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:34.257320 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 16:39:34.257473 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:34.257480 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:34.258373 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:34.258376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:34.259527 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:34.259538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:34.260129 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:34.260133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:34.260240 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:34.260292 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 16:39:34.333896 (+ 73604us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 16:39:34.334792 (+ 896us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.71.126:45583: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":142,"threads_started":1}
I20260928 16:39:34.335974 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 31347
2026-09-28T16:39:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (389 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (152 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (152 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:39:34 dist-test-slave-nqgh krb5kdc[31456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:39:34 dist-test-slave-nqgh krb5kdc[31456](info): set up 2 sockets
Sep 28 16:39:34 dist-test-slave-nqgh krb5kdc[31456](info): commencing operation
krb5kdc: starting...
W20260928 16:39:35.616626 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:39:35 dist-test-slave-nqgh krb5kdc[31456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:39:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:39:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:39:35.758973 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:45881
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:45881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:35.882498 31469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:35.882882 31469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:35.882927 31469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:35.887269 31469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:35.887351 31469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:35.887369 31469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:35.887384 31469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:35.887399 31469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:39:35.893074 31469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:45881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:45881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:35.894191 31469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:35.895241 31469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:35.904968 31474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:35.904980 31477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:35.905066 31469 server_base.cc:1034] running on GCE node
W20260928 16:39:35.904980 31475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:35.906142 31469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:35.907727 31469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:35.908936 31469 hybrid_clock.cc:648] HybridClock initialized: now 1790613575908894 us; error 58 us; skew 500 ppm
Sep 28 16:39:35 dist-test-slave-nqgh krb5kdc[31456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613575, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:35.912603 31469 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:39:35.913914 31469 webserver.cc:460] Webserver started at http://127.25.71.126:43961/ using document root <none> and password file <none>
I20260928 16:39:35.914681 31469 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:35.914783 31469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:35.915037 31469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:35.917639 31469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "ec23c19da7304b6b941ac37f2501e0bd"
format_stamp: "Formatted at 2026-09-28 16:39:35 on dist-test-slave-nqgh"
server_key: "ae7e43d87a1d3c4e618d98150af4ad4c"
server_key_iv: "f82134de46b62daa36abecb6679dfc87"
server_key_version: "encryptionkey@0"
I20260928 16:39:35.918321 31469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "ec23c19da7304b6b941ac37f2501e0bd"
format_stamp: "Formatted at 2026-09-28 16:39:35 on dist-test-slave-nqgh"
server_key: "ae7e43d87a1d3c4e618d98150af4ad4c"
server_key_iv: "f82134de46b62daa36abecb6679dfc87"
server_key_version: "encryptionkey@0"
I20260928 16:39:35.923494 31469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 16:39:35.927199 31484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:35.929180 31469 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:35.929387 31469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "ec23c19da7304b6b941ac37f2501e0bd"
format_stamp: "Formatted at 2026-09-28 16:39:35 on dist-test-slave-nqgh"
server_key: "ae7e43d87a1d3c4e618d98150af4ad4c"
server_key_iv: "f82134de46b62daa36abecb6679dfc87"
server_key_version: "encryptionkey@0"
I20260928 16:39:35.929551 31469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:35.946287 31469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:35.950685 31469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:35.950944 31469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:35.960595 31469 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:45881
I20260928 16:39:35.960624 31536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:45881 every 8 connection(s)
I20260928 16:39:35.961894 31469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:39:35.965315 31537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:35.967379 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 31469
I20260928 16:39:35.967540 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:39:35.967864 25885 external_mini_cluster.cc:1428] Setting key 845469f2503716644ba7b23f20de8766
I20260928 16:39:35.972576 31537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd: Bootstrap starting.
I20260928 16:39:35.975524 31537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:35.976349 31537 log.cc:824] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:35.979308 31537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd: No bootstrap required, opened a new log
Sep 28 16:39:35 dist-test-slave-nqgh krb5kdc[31456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:35.983527 31537 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec23c19da7304b6b941ac37f2501e0bd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 45881 } }
I20260928 16:39:35.983852 31537 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:35.983897 31537 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec23c19da7304b6b941ac37f2501e0bd, State: Initialized, Role: FOLLOWER
I20260928 16:39:35.984392 31537 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec23c19da7304b6b941ac37f2501e0bd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 45881 } }
I20260928 16:39:35.984584 31537 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:39:35.984660 31537 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:39:35.984776 31537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:35.986055 31537 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec23c19da7304b6b941ac37f2501e0bd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 45881 } }
I20260928 16:39:35.986471 31537 leader_election.cc:302] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec23c19da7304b6b941ac37f2501e0bd; no voters:
I20260928 16:39:35.986877 31537 leader_election.cc:288] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:39:35.987149 31542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:35.987494 31542 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd [term 1 LEADER]: Becoming Leader. State: Replica: ec23c19da7304b6b941ac37f2501e0bd, State: Running, Role: LEADER
I20260928 16:39:35.987747 31540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:35.969948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50360 (local address 127.25.71.126:45881)
0928 16:39:35.970583 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:35.970605 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:35.970686 (+ 81us) server_negotiation.cc:407] Connection header received
0928 16:39:35.971562 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:35.971588 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:35.972082 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:35.972511 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:35.973674 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:35.975292 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:35.976460 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:35.977111 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:35.977288 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:35.980554 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:35.980561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:35.980581 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:35.980621 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:35.983583 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:35.984455 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:35.984462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:35.984469 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:35.984647 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:35.985053 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:35.985056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:35.985058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:35.985617 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:35.985830 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:35.986308 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 16:39:35.986691 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":187,"threads_started":1}
I20260928 16:39:35.987962 31542 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec23c19da7304b6b941ac37f2501e0bd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 45881 } }
I20260928 16:39:35.988360 31537 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:35.989753 31544 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec23c19da7304b6b941ac37f2501e0bd. Latest consensus state: current_term: 1 leader_uuid: "ec23c19da7304b6b941ac37f2501e0bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec23c19da7304b6b941ac37f2501e0bd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 45881 } } }
I20260928 16:39:35.989938 31544 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:35.990128 31543 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec23c19da7304b6b941ac37f2501e0bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec23c19da7304b6b941ac37f2501e0bd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 45881 } } }
I20260928 16:39:35.990257 31543 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd [sys.catalog]: This master's current role is: LEADER
W20260928 16:39:35.993501 31557 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:39:35.993582 31557 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:39:35.993665 31551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:39:35.995276 31551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:39:36.001847 31551 catalog_manager.cc:1347] Generated new cluster ID: ee5f13600b724e04b14eb7dfd9f05cdc
I20260928 16:39:36.001971 31551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:39:36.014061 31551 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:39:36.015759 31551 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:39:36.032681 31551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec23c19da7304b6b941ac37f2501e0bd: Generated new TSK 0
I20260928 16:39:36.034004 31551 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
I20260928 16:39:36.074980 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:45881
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:36.198343 31563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:36.198726 31563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:36.198782 31563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:36.203244 31563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:36.203356 31563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:36.203444 31563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:39:36.209684 31563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:45881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:36.210870 31563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:36.211935 31563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:39:36.223433 31563 server_base.cc:1034] running on GCE node
W20260928 16:39:36.223358 31568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:36.223368 31571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:36.223397 31569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:36.224326 31563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:36.225124 31563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:36.226333 31563 hybrid_clock.cc:648] HybridClock initialized: now 1790613576226299 us; error 59 us; skew 500 ppm
Sep 28 16:39:36 dist-test-slave-nqgh krb5kdc[31456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613576, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:36.229786 31563 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:39:36.231238 31563 webserver.cc:460] Webserver started at http://127.25.71.65:35595/ using document root <none> and password file <none>
I20260928 16:39:36.231938 31563 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:36.232020 31563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:36.232241 31563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:36.234779 31563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "bbce31beec6842988d878267c8e9e225"
format_stamp: "Formatted at 2026-09-28 16:39:36 on dist-test-slave-nqgh"
server_key: "4588f63640a08c6b217650ef99cbd7c8"
server_key_iv: "68c32328c583246826a29acf16351bc6"
server_key_version: "encryptionkey@0"
I20260928 16:39:36.235445 31563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "bbce31beec6842988d878267c8e9e225"
format_stamp: "Formatted at 2026-09-28 16:39:36 on dist-test-slave-nqgh"
server_key: "4588f63640a08c6b217650ef99cbd7c8"
server_key_iv: "68c32328c583246826a29acf16351bc6"
server_key_version: "encryptionkey@0"
I20260928 16:39:36.240098 31563 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:36.243505 31578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:36.245193 31563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:39:36.245357 31563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "bbce31beec6842988d878267c8e9e225"
format_stamp: "Formatted at 2026-09-28 16:39:36 on dist-test-slave-nqgh"
server_key: "4588f63640a08c6b217650ef99cbd7c8"
server_key_iv: "68c32328c583246826a29acf16351bc6"
server_key_version: "encryptionkey@0"
I20260928 16:39:36.245492 31563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:36.267674 31563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:36.271701 31563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:36.271984 31563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:36.272652 31563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:36.273561 31563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:36.273612 31563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:36.273666 31563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:36.273694 31563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:36.283692 31563 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:36743
I20260928 16:39:36.283767 31691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:36743 every 8 connection(s)
I20260928 16:39:36.284782 31563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:39:36.294600 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 31563
I20260928 16:39:36.294842 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:39:36.295197 25885 external_mini_cluster.cc:1428] Setting key 6fa2dc1c6a8aa6410b5c7ac5b3e1fde2
Sep 28 16:39:36 dist-test-slave-nqgh krb5kdc[31456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613576, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:36.302929 31540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:36.287387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:40885 (local address 127.25.71.126:45881)
0928 16:39:36.287597 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:36.287603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:36.288725 (+ 1122us) server_negotiation.cc:407] Connection header received
0928 16:39:36.290522 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:36.290526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:36.290595 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:36.290750 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:36.292494 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:36.293099 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:36.294572 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:36.294984 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:36.295117 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:36.298756 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:36.298761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:36.298763 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:36.298797 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:36.300362 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:36.301007 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:36.301011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:36.301012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:36.301099 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:36.301489 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:36.301493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:36.301495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:36.301734 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:36.301819 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:36.302485 (+ 666us) server_negotiation.cc:299] Negotiation successful
0928 16:39:36.302700 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 16:39:36.304075 31694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:36.287797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:45881 (local address 127.25.71.65:40885)
0928 16:39:36.288509 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:36.288579 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:36.290214 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:36.291000 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:36.291010 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:36.291580 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:36.292269 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:36.292287 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:36.293283 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:36.293286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:36.294389 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:36.294401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:36.295186 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:36.295219 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:36.295445 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:36.296310 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:36.296335 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:36.298472 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:36.300517 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:36.300525 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:36.300549 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:36.300900 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:36.301207 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:36.301211 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:36.301213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:36.301395 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:36.301850 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:36.301859 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:36.302223 (+ 364us) client_negotiation.cc:769] Sending connection context
0928 16:39:36.302431 (+ 208us) client_negotiation.cc:240] Negotiation successful
0928 16:39:36.302955 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":170,"threads_started":1}
WARNING: no policy specified for kudu/127.25.71.66@KRBTEST.COM; defaulting to no policy
I20260928 16:39:36.305909 31692 heartbeater.cc:347] Connected to a master server at 127.25.71.126:45881
I20260928 16:39:36.306288 31692 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:36.306995 31692 heartbeater.cc:511] Master 127.25.71.126:45881 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.66@KRBTEST.COM" created.
I20260928 16:39:36.309072 31493 ts_manager.cc:194] Registered new tserver with Master: bbce31beec6842988d878267c8e9e225 (127.25.71.65:36743)
I20260928 16:39:36.310314 31493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:40885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
I20260928 16:39:36.327142 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:45881
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:36.442871 31697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:36.443189 31697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:36.443254 31697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:36.447556 31697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:36.447645 31697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:36.447734 31697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:39:36.453524 31697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:45881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:36.454829 31697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:36.455899 31697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:36.467013 31705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:36.466948 31703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:36.467064 31702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:36.467677 31697 server_base.cc:1034] running on GCE node
I20260928 16:39:36.468309 31697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:36.469250 31697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:36.470465 31697 hybrid_clock.cc:648] HybridClock initialized: now 1790613576470444 us; error 44 us; skew 500 ppm
Sep 28 16:39:36 dist-test-slave-nqgh krb5kdc[31456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613576, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:36.474736 31697 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:39:36.476187 31697 webserver.cc:460] Webserver started at http://127.25.71.66:37435/ using document root <none> and password file <none>
I20260928 16:39:36.476992 31697 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:36.477087 31697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:36.477375 31697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:36.480315 31697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "40b9d55c111648768331f3251332f7a0"
format_stamp: "Formatted at 2026-09-28 16:39:36 on dist-test-slave-nqgh"
server_key: "237775f7b8ac75ab17b6d1f02987e564"
server_key_iv: "12ec414b2fdc3b7cc000ae9aae2cacf2"
server_key_version: "encryptionkey@0"
I20260928 16:39:36.481088 31697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "40b9d55c111648768331f3251332f7a0"
format_stamp: "Formatted at 2026-09-28 16:39:36 on dist-test-slave-nqgh"
server_key: "237775f7b8ac75ab17b6d1f02987e564"
server_key_iv: "12ec414b2fdc3b7cc000ae9aae2cacf2"
server_key_version: "encryptionkey@0"
I20260928 16:39:36.486552 31697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:39:36.490612 31712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:36.492215 31697 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:36.492365 31697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "40b9d55c111648768331f3251332f7a0"
format_stamp: "Formatted at 2026-09-28 16:39:36 on dist-test-slave-nqgh"
server_key: "237775f7b8ac75ab17b6d1f02987e564"
server_key_iv: "12ec414b2fdc3b7cc000ae9aae2cacf2"
server_key_version: "encryptionkey@0"
I20260928 16:39:36.492501 31697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:36.512073 31697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:36.515676 31697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:36.515856 31697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:36.516453 31697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:36.517414 31697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:36.517473 31697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:36.517518 31697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:36.517546 31697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:36.527688 31697 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:42347
I20260928 16:39:36.527720 31826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:42347 every 8 connection(s)
I20260928 16:39:36.528641 31697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:39:36.535847 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 31697
I20260928 16:39:36.536006 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:39:36.536355 25885 external_mini_cluster.cc:1428] Setting key 095d5fdd92865f813d9cfbda03adcf4e
Sep 28 16:39:36 dist-test-slave-nqgh krb5kdc[31456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613576, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:36.545977 31540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:36.530964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:43031 (local address 127.25.71.126:45881)
0928 16:39:36.531096 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:36.531100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:36.532380 (+ 1280us) server_negotiation.cc:407] Connection header received
0928 16:39:36.534203 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:36.534207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:36.534270 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:36.534349 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:36.535885 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:36.536710 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:36.538350 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:36.538683 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:36.538779 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:36.542373 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:36.542376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:36.542378 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:36.542414 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:36.543797 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:36.544403 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:36.544406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:36.544407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:36.544487 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:36.544811 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:36.544814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:36.544816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:36.545024 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:36.545109 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:36.545656 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 16:39:36.545802 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:39:36.546964 31829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:36.531443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:45881 (local address 127.25.71.66:43031)
0928 16:39:36.532179 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:36.532245 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:36.533844 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:36.534566 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:36.534574 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:36.535036 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:36.535663 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:36.535683 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:36.536970 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:36.536974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:36.538186 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:36.538203 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:36.538879 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:36.538884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:36.539154 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:36.539944 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:36.539970 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:36.542149 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:36.543942 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:36.543950 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:36.543968 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:36.544296 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:36.544570 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:36.544574 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:36.544576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:36.544724 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:36.545122 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:36.545131 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:36.545415 (+ 284us) client_negotiation.cc:769] Sending connection context
0928 16:39:36.545599 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 16:39:36.545987 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":184,"threads_started":1}
WARNING: no policy specified for kudu/127.25.71.67@KRBTEST.COM; defaulting to no policy
I20260928 16:39:36.548687 31827 heartbeater.cc:347] Connected to a master server at 127.25.71.126:45881
I20260928 16:39:36.549016 31827 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:36.549893 31827 heartbeater.cc:511] Master 127.25.71.126:45881 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.67@KRBTEST.COM" created.
I20260928 16:39:36.551432 31493 ts_manager.cc:194] Registered new tserver with Master: 40b9d55c111648768331f3251332f7a0 (127.25.71.66:42347)
I20260928 16:39:36.552160 31493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:43031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
I20260928 16:39:36.570497 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:45881
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:36.708024 31832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:36.708460 31832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:36.708514 31832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:36.713300 31832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:36.713418 31832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:36.713491 31832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:39:36.720026 31832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:45881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:36.721645 31832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:36.723058 31832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:36.735052 31838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:36.735044 31840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:36.735227 31837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:36.735940 31832 server_base.cc:1034] running on GCE node
I20260928 16:39:36.736454 31832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:36.737274 31832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:36.738489 31832 hybrid_clock.cc:648] HybridClock initialized: now 1790613576738447 us; error 69 us; skew 500 ppm
Sep 28 16:39:36 dist-test-slave-nqgh krb5kdc[31456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613576, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:36.741760 31832 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:39:36.742861 31832 webserver.cc:460] Webserver started at http://127.25.71.67:41713/ using document root <none> and password file <none>
I20260928 16:39:36.743531 31832 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:36.743604 31832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:36.743815 31832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:36.746105 31832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "949068c8e59d49e0a78097683e2f29b8"
format_stamp: "Formatted at 2026-09-28 16:39:36 on dist-test-slave-nqgh"
server_key: "cbe428b6a6f05128e94fea7826abbaae"
server_key_iv: "c2fdb9f1020a247f5dd8faa441993843"
server_key_version: "encryptionkey@0"
I20260928 16:39:36.746735 31832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "949068c8e59d49e0a78097683e2f29b8"
format_stamp: "Formatted at 2026-09-28 16:39:36 on dist-test-slave-nqgh"
server_key: "cbe428b6a6f05128e94fea7826abbaae"
server_key_iv: "c2fdb9f1020a247f5dd8faa441993843"
server_key_version: "encryptionkey@0"
I20260928 16:39:36.751469 31832 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260928 16:39:36.754726 31847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:36.756377 31832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:39:36.756541 31832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "949068c8e59d49e0a78097683e2f29b8"
format_stamp: "Formatted at 2026-09-28 16:39:36 on dist-test-slave-nqgh"
server_key: "cbe428b6a6f05128e94fea7826abbaae"
server_key_iv: "c2fdb9f1020a247f5dd8faa441993843"
server_key_version: "encryptionkey@0"
I20260928 16:39:36.756664 31832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:36.775307 31832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:36.778604 31832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:36.778824 31832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:36.779461 31832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:36.780421 31832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:36.780476 31832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:36.780521 31832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:36.780552 31832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:36.791450 31832 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:40117
I20260928 16:39:36.791529 31962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:40117 every 8 connection(s)
I20260928 16:39:36.792730 31832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:39:36.800151 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 31832
I20260928 16:39:36.800357 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:39:36.800684 25885 external_mini_cluster.cc:1428] Setting key e1ce029c8cda7b02c365c0520c819084
Sep 28 16:39:36 dist-test-slave-nqgh krb5kdc[31456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613576, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:36.809463 31540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:36.795197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:51037 (local address 127.25.71.126:45881)
0928 16:39:36.795401 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:36.795407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:36.796562 (+ 1155us) server_negotiation.cc:407] Connection header received
0928 16:39:36.798153 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:36.798156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:36.798220 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:36.798298 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:36.799673 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:36.800253 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:36.801587 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:36.801821 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:36.801941 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:36.805379 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:36.805383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:36.805385 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:36.805427 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:36.806936 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:36.807647 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:36.807650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:36.807652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:36.807758 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:36.808151 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:36.808153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:36.808155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:36.808380 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:36.808472 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:36.809170 (+ 698us) server_negotiation.cc:299] Negotiation successful
0928 16:39:36.809314 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 16:39:36.810876 31965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:36.795668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:45881 (local address 127.25.71.67:51037)
0928 16:39:36.796352 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:36.796420 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:36.797909 (+ 1489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:36.798514 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:36.798523 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:36.798971 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:36.799514 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:36.799528 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:36.800390 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:36.800394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:36.801462 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:36.801474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:36.802061 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:36.802068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:36.802373 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:36.803206 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:36.803234 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:36.805120 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:36.807087 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:36.807094 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:36.807115 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:36.807512 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:36.807854 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:36.807856 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:36.807857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:36.808026 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:36.808499 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:36.808506 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:36.808879 (+ 373us) client_negotiation.cc:769] Sending connection context
0928 16:39:36.809086 (+ 207us) client_negotiation.cc:240] Negotiation successful
0928 16:39:36.809469 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":171,"threads_started":1}
I20260928 16:39:36.812516 31963 heartbeater.cc:347] Connected to a master server at 127.25.71.126:45881
I20260928 16:39:36.812858 31963 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:36.813524 31963 heartbeater.cc:511] Master 127.25.71.126:45881 requested a full tablet report, sending...
I20260928 16:39:36.815088 31501 ts_manager.cc:194] Registered new tserver with Master: 949068c8e59d49e0a78097683e2f29b8 (127.25.71.67:40117)
I20260928 16:39:36.816073 31501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:51037
I20260928 16:39:36.816354 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/test-admin.keytab.
Sep 28 16:39:36 dist-test-slave-nqgh krb5kdc[31456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:36.837752 25885 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 16:39:36.841549 31970 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:36.839651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.25.71.65:36743)
0928 16:39:36.840071 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:36.840079 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:36.840111 (+ 32us) server_negotiation.cc:407] Connection header received
0928 16:39:36.840170 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:36.840176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:36.840382 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:36.840523 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:36.841361 (+ 838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55090: BlockingRecv error: recv got EOF from 127.0.0.1:55090 (error 108)
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":135,"threads_started":1}
W20260928 16:39:36.842212 25885 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/krb5cc' of type FILE
Sep 28 16:39:36 dist-test-slave-nqgh krb5kdc[31456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:36.843568 25885 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 16:39:36 dist-test-slave-nqgh krb5kdc[31456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:36.844601 25885 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 16:39:36 dist-test-slave-nqgh krb5kdc[31456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.65@KRBTEST.COM
I20260928 16:39:36.857560 31970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:36.845910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55098 (local address 127.25.71.65:36743)
0928 16:39:36.846079 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:36.846085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:36.846174 (+ 89us) server_negotiation.cc:407] Connection header received
0928 16:39:36.846342 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:36.846346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:36.846402 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:36.846477 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:36.847190 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:36.848060 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:36.848999 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:36.849569 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:36.849682 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:36.852821 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:36.852839 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:36.852860 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:36.852904 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:36.855029 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:36.855771 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:36.855779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:36.855780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:36.855915 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:36.856272 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:36.856275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:36.856276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:36.856652 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:36.856894 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:36.857201 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 16:39:36.857393 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 16:39:36.863520 31970 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:36.861974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55112 (local address 127.25.71.65:36743)
0928 16:39:36.862138 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:36.862142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:36.862325 (+ 183us) server_negotiation.cc:407] Connection header received
0928 16:39:36.862518 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:36.862522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:36.862578 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:36.862750 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:36.863361 (+ 611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55112: BlockingRecv error: recv got EOF from 127.0.0.1:55112 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 16:39:36.863583 25885 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/krb5cc' of type FILE
Sep 28 16:39:36 dist-test-slave-nqgh krb5kdc[31456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:36.864811 25885 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 16:39:36 dist-test-slave-nqgh krb5kdc[31456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:36.865823 25885 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 16:39:36 dist-test-slave-nqgh krb5kdc[31456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.65@KRBTEST.COM
I20260928 16:39:36.876992 31970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:36.867326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55114 (local address 127.25.71.65:36743)
0928 16:39:36.867510 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:36.867515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:36.867540 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:39:36.867748 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:36.867751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:36.867813 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:36.868018 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:36.868672 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:36.869259 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:36.870256 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:36.870543 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:36.870707 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:36.873493 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:36.873496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:36.873499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:36.873532 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:36.875046 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:36.875586 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:36.875590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:36.875591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:36.875680 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:36.876008 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:36.876011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:36.876013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:36.876229 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:36.876382 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:36.876674 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 16:39:36.876839 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 16:39:36.880939 31970 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:36.879590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55120 (local address 127.25.71.65:36743)
0928 16:39:36.879750 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:36.879755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:36.879912 (+ 157us) server_negotiation.cc:407] Connection header received
0928 16:39:36.880062 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:36.880065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:36.880203 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:36.880285 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:36.880810 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55120: BlockingRecv error: recv got EOF from 127.0.0.1:55120 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 16:39:36.881336 25885 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613545667819-25885-0/krb5kdc/krb5cc' of type FILE
Sep 28 16:39:36 dist-test-slave-nqgh krb5kdc[31456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:36.882570 25885 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 16:39:36 dist-test-slave-nqgh krb5kdc[31456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:36.883584 25885 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 16:39:36 dist-test-slave-nqgh krb5kdc[31456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.65@KRBTEST.COM
I20260928 16:39:36.895381 31970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:36.884729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55136 (local address 127.25.71.65:36743)
0928 16:39:36.884899 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:36.884904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:36.885124 (+ 220us) server_negotiation.cc:407] Connection header received
0928 16:39:36.885219 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:36.885222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:36.885277 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:36.885354 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:36.886124 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:36.886869 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:36.887788 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:36.888081 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:36.888185 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:36.891263 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:36.891268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:36.891270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:36.891310 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:36.893044 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:36.893726 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:36.893729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:36.893730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:36.893839 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:36.894291 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:36.894294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:36.894295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:36.894514 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:36.894712 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:36.895073 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 16:39:36.895213 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:39:36.897063 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 31563
I20260928 16:39:36.903266 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 31697
I20260928 16:39:36.909956 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 31832
I20260928 16:39:36.916661 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 31469
2026-09-28T16:39:36Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2277 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:39:36 dist-test-slave-nqgh krb5kdc[31983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:39:36 dist-test-slave-nqgh krb5kdc[31983](info): set up 2 sockets
Sep 28 16:39:36 dist-test-slave-nqgh krb5kdc[31983](info): commencing operation
krb5kdc: starting...
W20260928 16:39:37.901088 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:39:37 dist-test-slave-nqgh krb5kdc[31983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:39:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:39:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.126.keytab.
Entry for principal oryx/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.126.keytab.
I20260928 16:39:38.039819 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:33855
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.126.keytab
--principal=oryx/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:33855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:38.164039 31996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:38.164395 31996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:38.164441 31996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:38.168821 31996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:38.168910 31996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:38.168929 31996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:38.168946 31996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:38.168960 31996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:39:38.174612 31996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:33855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.126.keytab
--principal=oryx/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:33855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:38.175820 31996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:38.176940 31996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:38.187116 32002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:38.187301 31996 server_base.cc:1034] running on GCE node
W20260928 16:39:38.187125 32001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:38.187156 32004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:38.188721 31996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:38.190095 31996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:38.191321 31996 hybrid_clock.cc:648] HybridClock initialized: now 1790613578191250 us; error 109 us; skew 500 ppm
Sep 28 16:39:38 dist-test-slave-nqgh krb5kdc[31983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613578, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:38.194875 31996 init.cc:377] Logged in from keytab as oryx/127.25.71.126@KRBTEST.COM (short username oryx)
I20260928 16:39:38.196009 31996 webserver.cc:460] Webserver started at http://127.25.71.126:40399/ using document root <none> and password file <none>
I20260928 16:39:38.196640 31996 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:38.196714 31996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:38.196925 31996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:38.199178 31996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "f350c795eedf4b9a91b21283d5e0f6d9"
format_stamp: "Formatted at 2026-09-28 16:39:38 on dist-test-slave-nqgh"
server_key: "b57a71f648b6d654422e2f3eadbdd915"
server_key_iv: "401c4a60bcc74fce46083e99dee9e8c8"
server_key_version: "encryptionkey@0"
I20260928 16:39:38.199757 31996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "f350c795eedf4b9a91b21283d5e0f6d9"
format_stamp: "Formatted at 2026-09-28 16:39:38 on dist-test-slave-nqgh"
server_key: "b57a71f648b6d654422e2f3eadbdd915"
server_key_iv: "401c4a60bcc74fce46083e99dee9e8c8"
server_key_version: "encryptionkey@0"
I20260928 16:39:38.204197 31996 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:38.207464 32011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:38.209224 31996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:39:38.209410 31996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "f350c795eedf4b9a91b21283d5e0f6d9"
format_stamp: "Formatted at 2026-09-28 16:39:38 on dist-test-slave-nqgh"
server_key: "b57a71f648b6d654422e2f3eadbdd915"
server_key_iv: "401c4a60bcc74fce46083e99dee9e8c8"
server_key_version: "encryptionkey@0"
I20260928 16:39:38.209539 31996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:38.230052 31996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:38.233847 31996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:38.234074 31996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:38.242226 31996 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:33855
I20260928 16:39:38.242579 32063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:33855 every 8 connection(s)
I20260928 16:39:38.243422 31996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:39:38.247359 32064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:38.247785 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 31996
I20260928 16:39:38.247920 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:39:38.248217 25885 external_mini_cluster.cc:1428] Setting key 9f505bdc629cfc7e680405148797f33f
I20260928 16:39:38.256688 32064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9: Bootstrap starting.
Sep 28 16:39:38 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM
I20260928 16:39:38.260789 32064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:38.261771 32064 log.cc:824] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:38.265952 32064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9: No bootstrap required, opened a new log
I20260928 16:39:38.266891 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:38.250057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.25.71.126:33855)
0928 16:39:38.250675 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:38.250701 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:38.250764 (+ 63us) server_negotiation.cc:407] Connection header received
0928 16:39:38.251664 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:38.251693 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:38.252294 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:38.252797 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:38.253709 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:38.255371 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:38.256402 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:38.257165 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:38.257357 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:38.261048 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:38.261056 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:38.261079 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:38.261130 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:38.263290 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:38.264034 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:38.264040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:38.264046 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:38.264192 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:38.264504 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:38.264506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:38.264507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:38.265023 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:38.265189 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:38.265551 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 16:39:38.265850 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":180,"threads_started":1}
I20260928 16:39:38.271075 32064 raft_consensus.cc:348] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } }
I20260928 16:39:38.271420 32064 raft_consensus.cc:374] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:38.271458 32064 raft_consensus.cc:729] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f350c795eedf4b9a91b21283d5e0f6d9, State: Initialized, Role: FOLLOWER
I20260928 16:39:38.272028 32064 consensus_queue.cc:260] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } }
I20260928 16:39:38.272204 32064 raft_consensus.cc:388] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:39:38.272276 32064 raft_consensus.cc:482] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:39:38.272392 32064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:38.273783 32064 raft_consensus.cc:504] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } }
I20260928 16:39:38.274314 32064 leader_election.cc:302] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f350c795eedf4b9a91b21283d5e0f6d9; no voters:
I20260928 16:39:38.274911 32064 leader_election.cc:288] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:39:38.275143 32069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:38.275624 32069 raft_consensus.cc:686] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [term 1 LEADER]: Becoming Leader. State: Replica: f350c795eedf4b9a91b21283d5e0f6d9, State: Running, Role: LEADER
I20260928 16:39:38.276103 32069 consensus_queue.cc:237] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } }
I20260928 16:39:38.276736 32064 sys_catalog.cc:564] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:38.278779 32070 sys_catalog.cc:455] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } }
I20260928 16:39:38.278831 32071 sys_catalog.cc:455] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f350c795eedf4b9a91b21283d5e0f6d9. Latest consensus state: current_term: 1 leader_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } }
I20260928 16:39:38.278995 32070 sys_catalog.cc:458] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:38.279021 32071 sys_catalog.cc:458] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [sys.catalog]: This master's current role is: LEADER
W20260928 16:39:38.282917 32084 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:39:38.283026 32084 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:39:38.283144 32083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:39:38.284054 32083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:39:38.291321 32083 catalog_manager.cc:1347] Generated new cluster ID: 521ae1f785a849e1a4661590fe3180c4
I20260928 16:39:38.291450 32083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:39:38.310930 32083 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:39:38.312474 32083 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:39:38.328220 32083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9: Generated new TSK 0
I20260928 16:39:38.329196 32083 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.25.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.65.keytab.
Entry for principal oryx/127.25.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.65.keytab.
I20260928 16:39:38.372164 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:33855
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.65.keytab
--principal=oryx/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:38.511432 32091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:38.511825 32091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:38.511878 32091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:38.516867 32091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:38.516981 32091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:38.517064 32091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:39:38.523576 32091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.65.keytab
--principal=oryx/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:33855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:38.525197 32091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:38.526564 32091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:38.538928 32096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:38.538921 32097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:38.538920 32099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:38.539355 32091 server_base.cc:1034] running on GCE node
I20260928 16:39:38.539870 32091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:38.540611 32091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:38.541803 32091 hybrid_clock.cc:648] HybridClock initialized: now 1790613578541778 us; error 49 us; skew 500 ppm
Sep 28 16:39:38 dist-test-slave-nqgh krb5kdc[31983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613578, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:38.545142 32091 init.cc:377] Logged in from keytab as oryx/127.25.71.65@KRBTEST.COM (short username oryx)
I20260928 16:39:38.546193 32091 webserver.cc:460] Webserver started at http://127.25.71.65:37095/ using document root <none> and password file <none>
I20260928 16:39:38.546859 32091 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:38.546936 32091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:38.547122 32091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:38.549369 32091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "a53637373b524375834e5110de5e7587"
format_stamp: "Formatted at 2026-09-28 16:39:38 on dist-test-slave-nqgh"
server_key: "945eacf6a85b09d920715a1a1ec9dfe6"
server_key_iv: "f21301591516f64df03ea4324bfa027f"
server_key_version: "encryptionkey@0"
I20260928 16:39:38.549981 32091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "a53637373b524375834e5110de5e7587"
format_stamp: "Formatted at 2026-09-28 16:39:38 on dist-test-slave-nqgh"
server_key: "945eacf6a85b09d920715a1a1ec9dfe6"
server_key_iv: "f21301591516f64df03ea4324bfa027f"
server_key_version: "encryptionkey@0"
I20260928 16:39:38.554601 32091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:39:38.558002 32107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:38.559793 32091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 16:39:38.560072 32091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "a53637373b524375834e5110de5e7587"
format_stamp: "Formatted at 2026-09-28 16:39:38 on dist-test-slave-nqgh"
server_key: "945eacf6a85b09d920715a1a1ec9dfe6"
server_key_iv: "f21301591516f64df03ea4324bfa027f"
server_key_version: "encryptionkey@0"
I20260928 16:39:38.560231 32091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:38.585804 32091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:38.589427 32091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:38.589608 32091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:38.590265 32091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:38.591231 32091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:38.591281 32091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:38.591329 32091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:38.591351 32091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:38.601441 32091 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:43829
I20260928 16:39:38.601511 32220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:43829 every 8 connection(s)
I20260928 16:39:38.602496 32091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:39:38.612115 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 32091
I20260928 16:39:38.612290 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:39:38.612687 25885 external_mini_cluster.cc:1428] Setting key be7486dc827123f30a5b703034e3f5cc
Sep 28 16:39:38 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613578, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM
I20260928 16:39:38.618654 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:38.605140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:38867 (local address 127.25.71.126:33855)
0928 16:39:38.605282 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:38.605286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:38.606492 (+ 1206us) server_negotiation.cc:407] Connection header received
0928 16:39:38.608104 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:38.608108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:38.608170 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:38.608245 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:38.609544 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:38.610104 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:38.611328 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:38.611573 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:38.611663 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:38.614733 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:38.614737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:38.614739 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:38.614770 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:38.616245 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:38.616793 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:38.616797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:38.616798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:38.616875 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:38.617219 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:38.617223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:38.617224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:38.617427 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:38.617519 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:38.618267 (+ 748us) server_negotiation.cc:299] Negotiation successful
0928 16:39:38.618460 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:39:38.619604 32223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:38.605633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.65:38867)
0928 16:39:38.606287 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:38.606350 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:38.607791 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:38.608447 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:38.608454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:38.608846 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:38.609381 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:38.609396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:38.610227 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:38.610230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:38.611206 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:38.611216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:38.611733 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:38.611737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:38.611937 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:38.612689 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:38.612706 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:38.614497 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:38.616380 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:38.616395 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:38.616417 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:38.616691 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:38.616969 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:38.616972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:38.616974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:38.617130 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:38.617539 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:38.617547 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:38.617832 (+ 285us) client_negotiation.cc:769] Sending connection context
0928 16:39:38.618057 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 16:39:38.618449 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":157,"threads_started":1}
I20260928 16:39:38.621467 32221 heartbeater.cc:347] Connected to a master server at 127.25.71.126:33855
I20260928 16:39:38.621829 32221 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:38.622558 32221 heartbeater.cc:511] Master 127.25.71.126:33855 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.25.71.66@KRBTEST.COM; defaulting to no policy
I20260928 16:39:38.624986 32028 ts_manager.cc:194] Registered new tserver with Master: a53637373b524375834e5110de5e7587 (127.25.71.65:43829)
I20260928 16:39:38.626489 32028 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:38867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.71.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.66.keytab.
Entry for principal oryx/127.25.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.66.keytab.
I20260928 16:39:38.647126 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:33855
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.66.keytab
--principal=oryx/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:38.771890 32226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:38.772186 32226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:38.772230 32226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:38.777231 32226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:38.777334 32226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:38.777426 32226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:39:38.783516 32226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.66.keytab
--principal=oryx/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:33855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:38.784735 32226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:38.785831 32226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:39:38.797331 32226 server_base.cc:1034] running on GCE node
W20260928 16:39:38.797292 32232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:38.797590 32234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:38.797312 32231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:38.798202 32226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:38.798995 32226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:38.800200 32226 hybrid_clock.cc:648] HybridClock initialized: now 1790613578800173 us; error 59 us; skew 500 ppm
Sep 28 16:39:38 dist-test-slave-nqgh krb5kdc[31983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613578, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:38.807367 32226 init.cc:377] Logged in from keytab as oryx/127.25.71.66@KRBTEST.COM (short username oryx)
I20260928 16:39:38.808493 32226 webserver.cc:460] Webserver started at http://127.25.71.66:42299/ using document root <none> and password file <none>
I20260928 16:39:38.809089 32226 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:38.809160 32226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:38.809358 32226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:38.811583 32226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "b0d7f25c092a41cc93edb7430142e0db"
format_stamp: "Formatted at 2026-09-28 16:39:38 on dist-test-slave-nqgh"
server_key: "c17f4125468c818e54bcb779611ade0d"
server_key_iv: "f4bb66c221cbd4f0db6c0875177b13b4"
server_key_version: "encryptionkey@0"
I20260928 16:39:38.812152 32226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "b0d7f25c092a41cc93edb7430142e0db"
format_stamp: "Formatted at 2026-09-28 16:39:38 on dist-test-slave-nqgh"
server_key: "c17f4125468c818e54bcb779611ade0d"
server_key_iv: "f4bb66c221cbd4f0db6c0875177b13b4"
server_key_version: "encryptionkey@0"
I20260928 16:39:38.816145 32226 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:39:38.818921 32241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:38.820726 32226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:39:38.820889 32226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "b0d7f25c092a41cc93edb7430142e0db"
format_stamp: "Formatted at 2026-09-28 16:39:38 on dist-test-slave-nqgh"
server_key: "c17f4125468c818e54bcb779611ade0d"
server_key_iv: "f4bb66c221cbd4f0db6c0875177b13b4"
server_key_version: "encryptionkey@0"
I20260928 16:39:38.821028 32226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:38.833613 32226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:38.837499 32226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:38.837721 32226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:38.838405 32226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:38.839947 32226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:38.840041 32226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:38.840116 32226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:38.840147 32226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:38.851807 32226 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:36905
I20260928 16:39:38.851915 32355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:36905 every 8 connection(s)
I20260928 16:39:38.853052 32226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:39:38.856194 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 32226
I20260928 16:39:38.856333 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:39:38.856715 25885 external_mini_cluster.cc:1428] Setting key eb556b0f6ca6aba47e969d534b30f427
Sep 28 16:39:38 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613578, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.25.71.67@KRBTEST.COM; defaulting to no policy
I20260928 16:39:38.870013 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:38.855548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:54929 (local address 127.25.71.126:33855)
0928 16:39:38.855719 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:38.855723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:38.857089 (+ 1366us) server_negotiation.cc:407] Connection header received
0928 16:39:38.858718 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:38.858721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:38.858775 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:38.858861 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:38.860579 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:38.861588 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:38.863066 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:38.863325 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:38.863423 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:38.866199 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:38.866202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:38.866204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:38.866233 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:38.867600 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:38.868176 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:38.868180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:38.868181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:38.868259 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:38.868593 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:38.868597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:38.868598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:38.868824 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:38.869118 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:38.869680 (+ 562us) server_negotiation.cc:299] Negotiation successful
0928 16:39:38.869832 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:39:38.871127 32358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:38.856113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.66:54929)
0928 16:39:38.856852 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:38.856932 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:38.858445 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:38.859096 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:38.859107 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:38.859635 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:38.860357 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:38.860377 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:38.861808 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:38.861812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:38.862899 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:38.862911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:38.863486 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:38.863490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:38.863663 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:38.864345 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:38.864363 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:38.865983 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:38.867730 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:38.867736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:38.867755 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:38.868077 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:38.868344 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:38.868347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:38.868349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:38.868501 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:38.869118 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:38.869126 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:38.869420 (+ 294us) client_negotiation.cc:769] Sending connection context
0928 16:39:38.869620 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 16:39:38.870005 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":177,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.71.67@KRBTEST.COM" created.
I20260928 16:39:38.872543 32356 heartbeater.cc:347] Connected to a master server at 127.25.71.126:33855
I20260928 16:39:38.872804 32356 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:38.873387 32356 heartbeater.cc:511] Master 127.25.71.126:33855 requested a full tablet report, sending...
I20260928 16:39:38.875111 32028 ts_manager.cc:194] Registered new tserver with Master: b0d7f25c092a41cc93edb7430142e0db (127.25.71.66:36905)
I20260928 16:39:38.876030 32028 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:54929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.67.keytab.
Entry for principal oryx/127.25.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.67.keytab.
I20260928 16:39:38.891279 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:33855
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.67.keytab
--principal=oryx/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:39.025674 32361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:39.026068 32361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:39.026119 32361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:39.030658 32361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:39.030761 32361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:39.030831 32361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:39:39.037217 32361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.67.keytab
--principal=oryx/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:33855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:39.038857 32361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:39.040257 32361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:39.052981 32366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:39.053030 32367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:39.053016 32369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:39.053177 32361 server_base.cc:1034] running on GCE node
I20260928 16:39:39.054003 32361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:39.055042 32361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:39.056254 32361 hybrid_clock.cc:648] HybridClock initialized: now 1790613579056216 us; error 49 us; skew 500 ppm
Sep 28 16:39:39 dist-test-slave-nqgh krb5kdc[31983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613579, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:39.060196 32361 init.cc:377] Logged in from keytab as oryx/127.25.71.67@KRBTEST.COM (short username oryx)
I20260928 16:39:39.061546 32361 webserver.cc:460] Webserver started at http://127.25.71.67:37883/ using document root <none> and password file <none>
I20260928 16:39:39.062290 32361 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:39.062373 32361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:39.062666 32361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:39.065456 32361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "83b2286be80340039d380a6bacdb8e43"
format_stamp: "Formatted at 2026-09-28 16:39:39 on dist-test-slave-nqgh"
server_key: "405a454c3ae59014a4b93b54e2e980f3"
server_key_iv: "db453fb7839975159b5f00d5e78fd2fb"
server_key_version: "encryptionkey@0"
I20260928 16:39:39.066133 32361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "83b2286be80340039d380a6bacdb8e43"
format_stamp: "Formatted at 2026-09-28 16:39:39 on dist-test-slave-nqgh"
server_key: "405a454c3ae59014a4b93b54e2e980f3"
server_key_iv: "db453fb7839975159b5f00d5e78fd2fb"
server_key_version: "encryptionkey@0"
I20260928 16:39:39.071341 32361 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 16:39:39.075230 32376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:39.077170 32361 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:39:39.077435 32361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "83b2286be80340039d380a6bacdb8e43"
format_stamp: "Formatted at 2026-09-28 16:39:39 on dist-test-slave-nqgh"
server_key: "405a454c3ae59014a4b93b54e2e980f3"
server_key_iv: "db453fb7839975159b5f00d5e78fd2fb"
server_key_version: "encryptionkey@0"
I20260928 16:39:39.077610 32361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:39.092207 32361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:39.096786 32361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:39.097065 32361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:39.097908 32361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:39.099143 32361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:39.099217 32361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:39.099290 32361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:39.099318 32361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:39.112104 32490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:44409 every 8 connection(s)
I20260928 16:39:39.112076 32361 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:44409
I20260928 16:39:39.113583 32361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:39:39.120414 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 32361
I20260928 16:39:39.120571 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:39:39.120900 25885 external_mini_cluster.cc:1428] Setting key 6a706f6610cfba3e8e93117ec8c3aad9
Sep 28 16:39:39 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613579, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM
I20260928 16:39:39.131160 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.116389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:47137 (local address 127.25.71.126:33855)
0928 16:39:39.116591 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:39.116596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:39.117968 (+ 1372us) server_negotiation.cc:407] Connection header received
0928 16:39:39.119893 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:39.119898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:39.120000 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:39.120150 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:39.121923 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.122564 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.123860 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.124121 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.124216 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.127304 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:39.127308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:39.127310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:39.127339 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:39.128927 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.129526 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.129530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.129532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.129611 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.129952 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.129956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.129958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.130158 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:39.130245 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:39.130836 (+ 591us) server_negotiation.cc:299] Negotiation successful
0928 16:39:39.130988 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 16:39:39.132054 32493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.116926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.67:47137)
0928 16:39:39.117664 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:39.117731 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:39.119484 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:39.120466 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:39.120475 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:39.121018 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:39.121733 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.121753 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.122731 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.122747 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.123714 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.123726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.124284 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.124296 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.124467 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.125157 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:39.125175 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:39.127072 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:39.129082 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.129090 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.129107 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.129416 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.129701 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.129704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.129706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.129860 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.130262 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:39.130270 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:39.130551 (+ 281us) client_negotiation.cc:769] Sending connection context
0928 16:39:39.130773 (+ 222us) client_negotiation.cc:240] Negotiation successful
0928 16:39:39.131139 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":187,"threads_started":1}
I20260928 16:39:39.134040 32491 heartbeater.cc:347] Connected to a master server at 127.25.71.126:33855
I20260928 16:39:39.134462 32491 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:39.135255 32491 heartbeater.cc:511] Master 127.25.71.126:33855 requested a full tablet report, sending...
I20260928 16:39:39.136803 32027 ts_manager.cc:194] Registered new tserver with Master: 83b2286be80340039d380a6bacdb8e43 (127.25.71.67:44409)
I20260928 16:39:39.137625 32027 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:47137
I20260928 16:39:39.150141 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.25.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.71.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.125.keytab.
Entry for principal oryx/127.25.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.125.keytab.
I20260928 16:39:39.184825 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.125:44555
--webserver_interface=127.25.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.125.keytab
--principal=oryx/127.25.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:33855,127.25.71.125:44555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:39.316447 32496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:39.316771 32496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:39.316823 32496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:39.321125 32496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:39.321218 32496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:39.321237 32496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:39.321254 32496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:39.321269 32496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:39:39.327279 32496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:33855,127.25.71.125:44555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.125.keytab
--principal=oryx/127.25.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.125:44555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.71.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
W20260928 16:39:39.327538 32496 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.25.71.126:33855,127.25.71.125:44555'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 16:39:39.328613 32496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:39.329756 32496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:39.339553 32501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:39.339615 32504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:39.339608 32502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:39.339903 32496 server_base.cc:1034] running on GCE node
I20260928 16:39:39.340610 32496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:39.341987 32496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:39.343273 32496 hybrid_clock.cc:648] HybridClock initialized: now 1790613579343251 us; error 46 us; skew 500 ppm
Sep 28 16:39:39 dist-test-slave-nqgh krb5kdc[31983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613579, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:39.347086 32496 init.cc:377] Logged in from keytab as oryx/127.25.71.125@KRBTEST.COM (short username oryx)
I20260928 16:39:39.348246 32496 webserver.cc:460] Webserver started at http://127.25.71.125:40227/ using document root <none> and password file <none>
I20260928 16:39:39.348908 32496 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:39.348985 32496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:39.349196 32496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:39.351478 32496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/data/instance:
uuid: "91facbd2cee44b1f814eb251c69daf0a"
format_stamp: "Formatted at 2026-09-28 16:39:39 on dist-test-slave-nqgh"
server_key: "db2d94a46fa9057fe53923f1ab8328bc"
server_key_iv: "ec3a92e14b126a2918d1185ab7baf6ba"
server_key_version: "encryptionkey@0"
I20260928 16:39:39.352062 32496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/wal/instance:
uuid: "91facbd2cee44b1f814eb251c69daf0a"
format_stamp: "Formatted at 2026-09-28 16:39:39 on dist-test-slave-nqgh"
server_key: "db2d94a46fa9057fe53923f1ab8328bc"
server_key_iv: "ec3a92e14b126a2918d1185ab7baf6ba"
server_key_version: "encryptionkey@0"
I20260928 16:39:39.356839 32496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:39:39.360352 32511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:39.362241 32496 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:39.362439 32496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/wal
uuid: "91facbd2cee44b1f814eb251c69daf0a"
format_stamp: "Formatted at 2026-09-28 16:39:39 on dist-test-slave-nqgh"
server_key: "db2d94a46fa9057fe53923f1ab8328bc"
server_key_iv: "ec3a92e14b126a2918d1185ab7baf6ba"
server_key_version: "encryptionkey@0"
I20260928 16:39:39.362574 32496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:39.390832 32496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:39.394702 32496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:39.395020 32496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:39.403331 32496 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.125:44555
I20260928 16:39:39.403667 32563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.125:44555 every 8 connection(s)
I20260928 16:39:39.404532 32496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/data/info.pb
I20260928 16:39:39.408702 32564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:39.414422 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 32496
I20260928 16:39:39.414013 32564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:39:39 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613579, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.125@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM
Sep 28 16:39:39 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.71.125@KRBTEST.COM
I20260928 16:39:39.430842 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.416742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.25.71.126:33855)
0928 16:39:39.416893 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:39.416897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:39.417585 (+ 688us) server_negotiation.cc:407] Connection header received
0928 16:39:39.419048 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:39.419052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:39.419131 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:39.419258 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:39.420387 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.420962 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.422244 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.422506 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.422590 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.426016 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:39.426022 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:39.426025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:39.426068 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:39.427768 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.428560 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.428564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.428566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.428665 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.429221 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.429225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.429227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.429595 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:39.429678 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:39.430481 (+ 803us) server_negotiation.cc:299] Negotiation successful
0928 16:39:39.430614 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:39:39.432113 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.416663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.0.0.1:56256)
0928 16:39:39.417398 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:39.417457 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:39.418663 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:39.419346 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:39.419352 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:39.419862 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:39.420184 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.420198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.421117 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.421121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.422077 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.422088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.422714 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.422718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.422925 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.423851 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:39.423870 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:39.425724 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:39.428013 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.428026 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.428040 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.428427 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.428805 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.428809 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.428811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.429074 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.429726 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:39.429736 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:39.430185 (+ 449us) client_negotiation.cc:769] Sending connection context
0928 16:39:39.430428 (+ 243us) client_negotiation.cc:240] Negotiation successful
0928 16:39:39.430936 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":53,"thread_start_us":196,"threads_started":1}
I20260928 16:39:39.432097 32568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.416663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46708 (local address 127.25.71.125:44555)
0928 16:39:39.417835 (+ 1172us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:39.417847 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:39.417891 (+ 44us) server_negotiation.cc:407] Connection header received
0928 16:39:39.418663 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:39.418697 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:39.419007 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:39.419187 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:39.420197 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.421270 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.422321 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.422861 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.422964 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.426529 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:39.426541 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:39.426564 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:39.426607 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:39.428604 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.429281 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.429289 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.429291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.429434 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.429856 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.429859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.429861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.430260 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:39.430477 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:39.430806 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 16:39:39.431417 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1043,"spinlock_wait_cycles":64128,"thread_start_us":196,"threads_started":1}
I20260928 16:39:39.433882 32564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:39:39 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613579, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.125@KRBTEST.COM for oryx/127.25.71.125@KRBTEST.COM
I20260928 16:39:39.444932 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.434755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:44555 (local address 127.0.0.1:46724)
0928 16:39:39.434909 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:39.434926 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:39.435031 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:39.435652 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:39.435655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:39.435851 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:39.435992 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.435998 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.436984 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.436987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.437623 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.437633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.438243 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.438246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.438359 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.439030 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:39.439043 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:39.440610 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:39.442817 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.442821 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.442823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.443072 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.443397 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.443400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.443401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.443551 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.444219 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:39.444222 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:39.444389 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 16:39:39.444499 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 16:39:39.444760 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 16:39:39.445142 32568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.435195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46724 (local address 127.25.71.125:44555)
0928 16:39:39.435383 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:39.435387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:39.435402 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:39:39.435448 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:39.435450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:39.435494 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:39.435584 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:39.436220 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.436811 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.437767 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.438048 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.438141 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.440838 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:39.440842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:39.440845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:39.440878 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:39.442585 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.443177 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.443180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.443182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.443272 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.443817 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.443820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.443822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.444079 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:39.444185 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:39.444622 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 16:39:39.444936 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":88}
I20260928 16:39:39.449281 32564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a: Bootstrap starting.
I20260928 16:39:39.451905 32564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:39.452692 32564 log.cc:824] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:39.455420 32564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a: No bootstrap required, opened a new log
I20260928 16:39:39.458612 32564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } }
I20260928 16:39:39.458925 32564 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:39.458976 32564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91facbd2cee44b1f814eb251c69daf0a, State: Initialized, Role: FOLLOWER
I20260928 16:39:39.459439 32564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } }
I20260928 16:39:39.460135 32570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } } }
I20260928 16:39:39.460341 32570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:39:39.460871 32564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [sys.catalog]: configured and running, proceeding with master startup.
W20260928 16:39:39.467120 32581 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:39:39.467258 32581 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:39:39.470300 32496 master_runner.cc:428] Detected that this master 91facbd2cee44b1f814eb251c69daf0a is joining an existing cluster
I20260928 16:39:39.470410 32496 master_runner.cc:432] Initiating AddMaster RPC to add 127.25.71.125:44555
I20260928 16:39:39.483867 32587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.473205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.0.0.1:56272)
0928 16:39:39.473847 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:39.473866 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:39.474073 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:39.474488 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:39.474490 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:39.474903 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:39.475186 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.475196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.476420 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.476423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.477948 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.477961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.478705 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.478709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.478788 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.479714 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:39.479728 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:39.480144 (+ 416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:39.482055 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.482059 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.482062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.482342 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.482650 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.482654 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.482656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.482757 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.483258 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:39.483261 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:39.483393 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 16:39:39.483486 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 16:39:39.483666 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":55,"spinlock_wait_cycles":3584,"thread_start_us":599,"threads_started":1}
I20260928 16:39:39.484197 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.473499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56272 (local address 127.25.71.126:33855)
0928 16:39:39.473658 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:39.473664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:39.473968 (+ 304us) server_negotiation.cc:407] Connection header received
0928 16:39:39.474202 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:39.474205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:39.474261 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:39.474308 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:39.475357 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.476229 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.478096 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.478491 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.478674 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.480307 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:39.480310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:39.480313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:39.480344 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:39.481912 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.482448 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.482451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.482452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.482545 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.482842 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.482845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.482847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.483105 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:39.483194 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:39.483647 (+ 453us) server_negotiation.cc:299] Negotiation successful
0928 16:39:39.483829 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:39:39.485311 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.473478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:44555 (local address 127.0.0.1:46738)
0928 16:39:39.473994 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:39.474005 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:39.474102 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:39.474401 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:39.474405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:39.474903 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:39.475186 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.475196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.477364 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.477368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.478235 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.478243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.478846 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.478849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.478926 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.479448 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:39.479461 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:39.480008 (+ 547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:39.482384 (+ 2376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.482387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.482389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.482712 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.483018 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.483021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.483023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.483095 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.484665 (+ 1570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:39.484668 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:39.484807 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 16:39:39.484925 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 16:39:39.485090 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":40,"spinlock_wait_cycles":5888,"thread_start_us":127,"threads_started":1}
I20260928 16:39:39.485363 32568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.473514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46738 (local address 127.25.71.125:44555)
0928 16:39:39.473656 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:39.473671 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:39.474097 (+ 426us) server_negotiation.cc:407] Connection header received
0928 16:39:39.474189 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:39.474193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:39.474254 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:39.474367 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:39.476440 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.477201 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.478337 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.478562 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.478702 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.480220 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:39.480226 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:39.480229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:39.480272 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:39.482208 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.482828 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.482833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.482835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.482914 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.483203 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.483206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.483208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.483421 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:39.483516 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:39.485065 (+ 1549us) server_negotiation.cc:299] Negotiation successful
0928 16:39:39.485208 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":54}
Sep 28 16:39:39 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613578, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.126@KRBTEST.COM for oryx/127.25.71.125@KRBTEST.COM
I20260928 16:39:39.501754 32568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.490304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46754 (local address 127.25.71.125:44555)
0928 16:39:39.490460 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:39.490465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:39.490772 (+ 307us) server_negotiation.cc:407] Connection header received
0928 16:39:39.491027 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:39.491031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:39.491102 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:39.491214 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:39.492083 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.492657 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.493981 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.494216 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.494302 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.497341 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:39.497345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:39.497347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:39.497382 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:39.499289 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.500000 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.500004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.500006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.500112 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.500429 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.500433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.500435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.500808 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:39.500926 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:39.501383 (+ 457us) server_negotiation.cc:299] Negotiation successful
0928 16:39:39.501545 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:39:39.501754 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.490195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:44555 (local address 127.0.0.1:46754)
0928 16:39:39.490596 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:39.490689 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:39.490844 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:39.491265 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:39.491271 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:39.491769 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:39.491936 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.491944 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.492778 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.492781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.493842 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.493855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.494337 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.494340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.494460 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.495216 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:39.495231 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:39.497133 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:39.499452 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.499465 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.499475 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.499872 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.500216 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.500219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.500222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.500329 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.500941 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:39.500949 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:39.501125 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 16:39:39.501312 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 16:39:39.501559 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for oryx/127.25.71.124@KRBTEST.COM; defaulting to no policy
I20260928 16:39:39.504002 32027 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.25.71.125:44555: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a"
member_type: NON_VOTER
last_known_addr {
host: "127.25.71.125"
port: 44555
}
attrs {
promote: true
}
}
dest_uuid: "f350c795eedf4b9a91b21283d5e0f6d9"
cas_config_opid_index: -1
I20260928 16:39:39.504905 32027 consensus_queue.cc:237] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.71.124@KRBTEST.COM" created.
I20260928 16:39:39.507592 32592 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 91facbd2cee44b1f814eb251c69daf0a (127.25.71.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true } } }
I20260928 16:39:39.508496 32538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:39.509989 32592 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.25.71.125:44555
I20260928 16:39:39.510331 32538 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 1 FOLLOWER]: Refusing update from remote peer f350c795eedf4b9a91b21283d5e0f6d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 16:39:39.510612 32593 sys_catalog.cc:455] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true } } }
I20260928 16:39:39.510756 32593 sys_catalog.cc:458] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:39.511216 32496 master.cc:537] Master@127.25.71.125:44555 shutting down...
I20260928 16:39:39.511900 32593 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 16:39:39.514598 32015 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 16:39:39.515765 32015 consensus_peers.cc:487] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 -> Peer 91facbd2cee44b1f814eb251c69daf0a (127.25.71.125:44555): Couldn't send request to peer 91facbd2cee44b1f814eb251c69daf0a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 16:39:39.517249 32496 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:39.517702 32496 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:39.517844 32496 tablet_replica.cc:330] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.124.keytab.
Entry for principal oryx/127.25.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.124.keytab.
I20260928 16:39:39.531198 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.124:38021
--webserver_interface=127.25.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.124.keytab
--principal=oryx/127.25.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:33855,127.25.71.125:44555,127.25.71.124:38021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:39:39.535961 32496 master.cc:552] Master@127.25.71.125:44555 shutdown complete.
I20260928 16:39:39.536065 32496 master_runner.cc:305] Clearing existing system tablet
I20260928 16:39:39.536419 32496 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:39.536496 32496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:39.538239 32496 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 16:39:39.539553 32599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:39.539959 32496 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:39.540122 32496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 16:39:39.540175 32496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/wal
uuid: "91facbd2cee44b1f814eb251c69daf0a"
format_stamp: "Formatted at 2026-09-28 16:39:39 on dist-test-slave-nqgh"
server_key: "db2d94a46fa9057fe53923f1ab8328bc"
server_key_iv: "ec3a92e14b126a2918d1185ab7baf6ba"
server_key_version: "encryptionkey@0"
I20260928 16:39:39.543735 32496 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:39.547363 32496 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 16:39:39.547523 32496 log.cc:1196] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 16:39:39.547829 32496 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a: Deleting consensus metadata
I20260928 16:39:39.548017 32496 master_runner.cc:315] Copying system tablet from 127.25.71.126:33855
I20260928 16:39:39.549585 32496 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.71.126:33855
I20260928 16:39:39.560276 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.550317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.0.0.1:56282)
0928 16:39:39.550771 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:39.550798 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:39.550935 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:39.551307 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:39.551311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:39.551550 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:39.551709 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.551717 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.553008 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.553013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.554017 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.554032 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.554741 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.554745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.554868 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.555557 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:39.555572 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:39.556026 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:39.558329 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.558333 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.558335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.558708 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.559080 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.559083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.559085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.559178 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.559669 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:39.559672 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:39.559794 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 16:39:39.559906 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 16:39:39.560067 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":124,"threads_started":1}
I20260928 16:39:39.560328 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.550482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56282 (local address 127.25.71.126:33855)
0928 16:39:39.550700 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:39.550706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:39.550893 (+ 187us) server_negotiation.cc:407] Connection header received
0928 16:39:39.551059 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:39.551063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:39.551131 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:39.551266 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:39.551869 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.552816 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.554184 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.554568 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.554724 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.556231 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:39.556236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:39.556239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:39.556274 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:39.558163 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.558845 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.558849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.558852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.558978 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.559282 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.559285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.559286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.559573 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:39.559678 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:39.559978 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 16:39:39.560135 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 16:39:39.561422 32048 tablet_copy_service.cc:140] P f350c795eedf4b9a91b21283d5e0f6d9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 91facbd2cee44b1f814eb251c69daf0a ({username='oryx', principal='oryx/127.25.71.125@KRBTEST.COM'} at 127.0.0.1:56282)
I20260928 16:39:39.561651 32048 tablet_copy_service.cc:161] P f350c795eedf4b9a91b21283d5e0f6d9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 91facbd2cee44b1f814eb251c69daf0a at {username='oryx', principal='oryx/127.25.71.125@KRBTEST.COM'} at 127.0.0.1:56282: session id = 91facbd2cee44b1f814eb251c69daf0a-00000000000000000000000000000000
I20260928 16:39:39.563767 32048 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 16:39:39.565976 32496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 16:39:39.569380 32496 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a: tablet copy: Starting download of 0 data blocks...
I20260928 16:39:39.569751 32496 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a: tablet copy: Starting download of 1 WAL segments...
I20260928 16:39:39.572794 32496 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 16:39:39.575524 32048 tablet_copy_service.cc:342] P f350c795eedf4b9a91b21283d5e0f6d9: Request end of tablet copy session 91facbd2cee44b1f814eb251c69daf0a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.25.71.125@KRBTEST.COM'} at 127.0.0.1:56282
I20260928 16:39:39.575778 32048 tablet_copy_service.cc:433] P f350c795eedf4b9a91b21283d5e0f6d9: ending tablet copy session 91facbd2cee44b1f814eb251c69daf0a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 91facbd2cee44b1f814eb251c69daf0a
I20260928 16:39:39.586900 32496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:39.589174 32606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:39.589221 32607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:39.589584 32609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:39.590298 32496 server_base.cc:1034] running on GCE node
I20260928 16:39:39.590582 32496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:39.591051 32496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:39.592281 32496 hybrid_clock.cc:648] HybridClock initialized: now 1790613579592248 us; error 52 us; skew 500 ppm
Sep 28 16:39:39 dist-test-slave-nqgh krb5kdc[31983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613579, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:39.594412 32496 init.cc:377] Logged in from keytab as oryx/127.25.71.125@KRBTEST.COM (short username oryx)
I20260928 16:39:39.595247 32496 webserver.cc:460] Webserver started at http://127.25.71.125:33821/ using document root <none> and password file <none>
I20260928 16:39:39.595480 32496 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:39.595527 32496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:39.597240 32496 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 16:39:39.598659 32616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:39.599017 32496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 16:39:39.599119 32496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/wal
uuid: "91facbd2cee44b1f814eb251c69daf0a"
format_stamp: "Formatted at 2026-09-28 16:39:39 on dist-test-slave-nqgh"
server_key: "db2d94a46fa9057fe53923f1ab8328bc"
server_key_iv: "ec3a92e14b126a2918d1185ab7baf6ba"
server_key_version: "encryptionkey@0"
I20260928 16:39:39.599216 32496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:39.620473 32496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:39.623085 32496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:39.628873 32496 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.125:44555
I20260928 16:39:39.629343 32668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.125:44555 every 8 connection(s)
I20260928 16:39:39.629680 32496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-1/data/info.pb
I20260928 16:39:39.629900 32221 heartbeater.cc:503] Master 127.25.71.126:33855 was elected leader, sending a full tablet report...
I20260928 16:39:39.633505 32669 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:39:39.634708 32669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a: Bootstrap starting.
I20260928 16:39:39.647998 32669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:39:39.648442 32669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a: Bootstrap complete.
I20260928 16:39:39.649303 32669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true } }
I20260928 16:39:39.649480 32669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91facbd2cee44b1f814eb251c69daf0a, State: Initialized, Role: LEARNER
I20260928 16:39:39.649612 32669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true } }
I20260928 16:39:39.650213 32669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:39.650141 32672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true } } }
I20260928 16:39:39.650321 32672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [sys.catalog]: This master's current role is: LEARNER
I20260928 16:39:39.654847 32683 catalog_manager.cc:1259] Loaded cluster ID: 521ae1f785a849e1a4661590fe3180c4
I20260928 16:39:39.654925 32683 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a: loading cluster ID for follower catalog manager: success
I20260928 16:39:39.656648 32683 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a: acquiring CA information for follower catalog manager: success
I20260928 16:39:39.657964 32683 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 16:39:39.697448 32597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:39.697853 32597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:39.697901 32597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:39.702696 32597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:39.702822 32597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:39.702843 32597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:39.702862 32597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:39.702877 32597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:39:39.709393 32597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:33855,127.25.71.125:44555,127.25.71.124:38021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.124.keytab
--principal=oryx/127.25.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.124:38021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.71.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:39.710979 32597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:39.712337 32597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:39.725282 32688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:39.725492 32597 server_base.cc:1034] running on GCE node
W20260928 16:39:39.725831 32689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:39.725335 32691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:39.727350 32597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:39.729387 32597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:39.732743 32597 hybrid_clock.cc:648] HybridClock initialized: now 1790613579732727 us; error 170 us; skew 500 ppm
Sep 28 16:39:39 dist-test-slave-nqgh krb5kdc[31983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613579, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:39.737106 32597 init.cc:377] Logged in from keytab as oryx/127.25.71.124@KRBTEST.COM (short username oryx)
I20260928 16:39:39.738862 32597 webserver.cc:460] Webserver started at http://127.25.71.124:45979/ using document root <none> and password file <none>
I20260928 16:39:39.739863 32597 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:39.739977 32597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:39.740321 32597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:39.744074 32597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/data/instance:
uuid: "2b1cd9df519e432b883536905633b131"
format_stamp: "Formatted at 2026-09-28 16:39:39 on dist-test-slave-nqgh"
server_key: "1c066cf26d3fbf2d4e00dc071194c371"
server_key_iv: "1775ed85161af0f2f2e91ef67120f564"
server_key_version: "encryptionkey@0"
I20260928 16:39:39.744995 32597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/wal/instance:
uuid: "2b1cd9df519e432b883536905633b131"
format_stamp: "Formatted at 2026-09-28 16:39:39 on dist-test-slave-nqgh"
server_key: "1c066cf26d3fbf2d4e00dc071194c371"
server_key_iv: "1775ed85161af0f2f2e91ef67120f564"
server_key_version: "encryptionkey@0"
I20260928 16:39:39.752916 32597 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.001s
I20260928 16:39:39.757537 32698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:39.759914 32597 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:39.760155 32597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/wal
uuid: "2b1cd9df519e432b883536905633b131"
format_stamp: "Formatted at 2026-09-28 16:39:39 on dist-test-slave-nqgh"
server_key: "1c066cf26d3fbf2d4e00dc071194c371"
server_key_iv: "1775ed85161af0f2f2e91ef67120f564"
server_key_version: "encryptionkey@0"
I20260928 16:39:39.760370 32597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:39.778718 32597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:39.783933 32597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:39.784150 32597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:39.791998 32597 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.124:38021
I20260928 16:39:39.792423 32750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.124:38021 every 8 connection(s)
I20260928 16:39:39.793185 32597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/data/info.pb
I20260928 16:39:39.797173 32751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:39.802599 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 32597
I20260928 16:39:39.802362 32751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:39:39 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613579, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.124@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM
Sep 28 16:39:39 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.71.124@KRBTEST.COM
I20260928 16:39:39.818267 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.805487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56298 (local address 127.25.71.126:33855)
0928 16:39:39.805648 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:39.805652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:39.805758 (+ 106us) server_negotiation.cc:407] Connection header received
0928 16:39:39.806659 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:39.806662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:39.806721 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:39.806849 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:39.808334 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.808975 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.810154 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.810384 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.810475 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.813494 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:39.813498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:39.813500 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:39.813561 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:39.815487 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.816294 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.816299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.816302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.816413 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.816890 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.816894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.816896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.817162 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:39.817284 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:39.817880 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 16:39:39.818067 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 16:39:39.819588 32754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.805204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.0.0.1:56298)
0928 16:39:39.805598 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:39.805651 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:39.806337 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:39.807424 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:39.807431 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:39.807897 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:39.808151 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.808167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.809122 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.809126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.810024 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.810035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.810551 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.810556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.810706 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.811401 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:39.811419 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:39.813282 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:39.815670 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.815681 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.815697 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.816139 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.816554 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.816558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.816560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.816756 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.817292 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:39.817303 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:39.817625 (+ 322us) client_negotiation.cc:769] Sending connection context
0928 16:39:39.817831 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 16:39:39.818299 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"spinlock_wait_cycles":3840,"thread_start_us":83,"threads_started":1}
I20260928 16:39:39.819586 32753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.804460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55330 (local address 127.25.71.124:38021)
0928 16:39:39.805067 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:39.805085 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:39.805141 (+ 56us) server_negotiation.cc:407] Connection header received
0928 16:39:39.806270 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:39.806297 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:39.806592 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:39.806909 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:39.807775 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.808989 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.809917 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.810411 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.810582 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.813807 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:39.813816 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:39.813832 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:39.813869 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:39.816097 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.816609 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.816615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.816616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.816766 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.817095 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.817098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.817100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.817611 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:39.817828 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:39.818078 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 16:39:39.818478 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":179,"server-negotiator.queue_time_us":414,"thread_start_us":159,"threads_started":1}
I20260928 16:39:39.821527 32751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:39:39 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613579, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.124@KRBTEST.COM for oryx/127.25.71.125@KRBTEST.COM
I20260928 16:39:39.831931 32754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.822303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:44555 (local address 127.0.0.1:46758)
0928 16:39:39.822441 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:39.822458 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:39.822567 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:39.823259 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:39.823261 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:39.823440 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:39.823588 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.823594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.824663 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.824667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.825284 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.825294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.825924 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.825933 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.826046 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.826595 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:39.826607 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:39.828223 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:39.830147 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.830150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.830152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.830448 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.830852 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.830855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.830856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.830958 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.831414 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:39.831416 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:39.831534 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 16:39:39.831621 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 16:39:39.831782 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 16:39:39.832084 32755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.822439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46758 (local address 127.25.71.125:44555)
0928 16:39:39.822953 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:39.822959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:39.822979 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:39:39.823036 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:39.823039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:39.823098 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:39.823193 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:39.823727 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.824518 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.825405 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.825688 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.825787 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.828457 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:39.828460 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:39.828463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:39.828495 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:39.829972 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.830560 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.830563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.830564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.830721 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.831095 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.831098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.831099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.831333 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:39.831412 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:39.831739 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 16:39:39.831890 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":119,"threads_started":1}
I20260928 16:39:39.833189 32751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:39:39 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613579, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.124@KRBTEST.COM for oryx/127.25.71.124@KRBTEST.COM
I20260928 16:39:39.841419 32753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.833925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55336 (local address 127.25.71.124:38021)
0928 16:39:39.834045 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:39.834049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:39.834063 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:39:39.834143 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:39.834146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:39.834184 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:39.834245 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:39.834652 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.835171 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.835880 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.836096 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.836180 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.838382 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:39.838385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:39.838387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:39.838406 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:39.839906 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.840325 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.840328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.840330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.840396 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.840622 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.840625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.840627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.840810 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:39.840893 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:39.841124 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 16:39:39.841277 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":57}
I20260928 16:39:39.841419 32754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.833773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:38021 (local address 127.0.0.1:55336)
0928 16:39:39.833902 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:39.833916 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:39.834051 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:39.834265 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:39.834267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:39.834440 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:39.834524 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.834530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.835277 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.835280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.835772 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.835779 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.836195 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.836199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.836266 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.836734 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:39.836745 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:39.838242 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:39.840058 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.840062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.840064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.840239 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.840473 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.840476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.840478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.840549 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.840893 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:39.840896 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:39.840984 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 16:39:39.841068 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 16:39:39.841225 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 16:39:39.845503 32751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: Bootstrap starting.
I20260928 16:39:39.847965 32751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:39.848722 32751 log.cc:824] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:39.851261 32751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: No bootstrap required, opened a new log
I20260928 16:39:39.854619 32751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } }
I20260928 16:39:39.854974 32751 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:39.855039 32751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b1cd9df519e432b883536905633b131, State: Initialized, Role: FOLLOWER
I20260928 16:39:39.855523 32751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } }
I20260928 16:39:39.856186 32757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } } }
I20260928 16:39:39.856441 32757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:39:39.856861 32751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 16:39:39.863355 300 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:39:39.863499 300 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:39:39.868225 32597 master_runner.cc:428] Detected that this master 2b1cd9df519e432b883536905633b131 is joining an existing cluster
I20260928 16:39:39.868352 32597 master_runner.cc:432] Initiating AddMaster RPC to add 127.25.71.124:38021
I20260928 16:39:39.875804 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 32091
I20260928 16:39:39.879253 32356 heartbeater.cc:503] Master 127.25.71.126:33855 was elected leader, sending a full tablet report...
I20260928 16:39:39.885941 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:43829
--local_ip_for_outbound_sockets=127.25.71.65
--tserver_master_addrs=127.25.71.126:33855
--webserver_port=37095
--webserver_interface=127.25.71.65
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.65.keytab
--principal=oryx/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.25.71.126:33855,127.25.71.125:44555
--tserver_master_addrs=127.25.71.126:33855,127.25.71.125:44555,127.25.71.124:38021 with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:39:39.888453 32753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.872175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55344 (local address 127.25.71.124:38021)
0928 16:39:39.872281 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:39.872285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:39.873010 (+ 725us) server_negotiation.cc:407] Connection header received
0928 16:39:39.873141 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:39.873144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:39.873183 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:39.873224 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:39.874324 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.875205 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.876898 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.877235 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.877659 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.884292 (+ 6633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:39.884298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:39.884302 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:39.884351 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:39.886304 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.886863 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.886866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.886868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.886941 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.887173 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.887176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.887178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.887421 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:39.887493 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:39.888046 (+ 553us) server_negotiation.cc:299] Negotiation successful
0928 16:39:39.888236 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":39,"spinlock_wait_cycles":1408}
I20260928 16:39:39.888343 307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.872118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:38021 (local address 127.0.0.1:55344)
0928 16:39:39.872931 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:39.872943 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:39.873057 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:39.873378 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:39.873381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:39.873668 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:39.874191 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.874202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.875330 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.875333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.876771 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.876783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.877860 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.877864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.877962 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.883392 (+ 5430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:39.883412 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:39.883995 (+ 583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:39.886461 (+ 2466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.886464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.886466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.886760 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.887026 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.887029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.887031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.887100 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.887550 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:39.887553 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:39.887639 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 16:39:39.887928 (+ 289us) client_negotiation.cc:240] Negotiation successful
0928 16:39:39.888110 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"spinlock_wait_cycles":16768,"thread_start_us":113,"threads_started":1}
I20260928 16:39:39.892874 32755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.871618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.25.71.125:44555)
0928 16:39:39.871781 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:39.871786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:39.872678 (+ 892us) server_negotiation.cc:407] Connection header received
0928 16:39:39.872834 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:39.872838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:39.872912 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:39.872980 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:39.884831 (+ 11851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.885683 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.886884 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.887217 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.887297 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.889098 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:39.889101 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:39.889104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:39.889131 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:39.891208 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.891626 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.891629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.891631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.891714 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.891958 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.891961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.891963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.892175 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:39.892263 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:39.892534 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 16:39:39.892687 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:39:39.892889 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.871917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:44555 (local address 127.0.0.1:46770)
0928 16:39:39.872557 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:39.872576 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:39.872713 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:39.884270 (+ 11557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:39.884273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:39.884464 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:39.884626 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.884633 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.885840 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.885844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.886734 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.886747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.887520 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.887523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.887626 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.888434 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:39.888448 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:39.888925 (+ 477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:39.891320 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.891324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.891327 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.891524 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.891798 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.891801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.891803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.891879 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.892264 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:39.892267 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:39.892383 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 16:39:39.892469 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 16:39:39.892656 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":160,"threads_started":1}
I20260928 16:39:39.898468 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.871609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56306 (local address 127.25.71.126:33855)
0928 16:39:39.871784 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:39.871789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:39.872541 (+ 752us) server_negotiation.cc:407] Connection header received
0928 16:39:39.872750 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:39.872752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:39.872809 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:39.872900 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:39.873699 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.874652 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.875842 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.876180 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.876256 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.895214 (+ 18958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:39.895220 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:39.895222 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:39.895260 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:39.896742 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.897201 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.897205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.897207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.897289 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.897545 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.897548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.897550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.897780 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:39.897868 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:39.898142 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 16:39:39.898273 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:39:39.898491 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.871460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.0.0.1:56306)
0928 16:39:39.872435 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:39.872449 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:39.872599 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:39.873075 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:39.873078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:39.873381 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:39.873558 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.873567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.874809 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.874812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.875711 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.875722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.876890 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.876894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.876997 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.877691 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:39.877725 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:39.895014 (+ 17289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:39.896876 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.896880 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.896883 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.897102 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.897378 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.897381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.897383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.897467 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.897869 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:39.897873 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:39.897978 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 16:39:39.898069 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 16:39:39.898274 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":3,"spinlock_wait_cycles":2944,"thread_start_us":162,"threads_started":1}
Sep 28 16:39:39 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613578, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.126@KRBTEST.COM for oryx/127.25.71.124@KRBTEST.COM
I20260928 16:39:39.911661 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.903219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:38021 (local address 127.0.0.1:55354)
0928 16:39:39.903430 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:39.903446 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:39.903539 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:39.903865 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:39.903868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:39.904095 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:39.904215 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.904223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.904980 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.904983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.905555 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.905562 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.905996 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.906000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.906120 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.906779 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:39.906792 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:39.908259 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:39.910027 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.910031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.910034 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.910257 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.910513 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.910517 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.910519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.910601 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.911063 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:39.911066 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:39.911171 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 16:39:39.911259 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 16:39:39.911453 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260928 16:39:39.911650 32753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.903350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55354 (local address 127.25.71.124:38021)
0928 16:39:39.903468 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:39.903473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:39.903486 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:39:39.903697 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:39.903700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:39.903755 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:39.903827 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:39.904329 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.904876 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.905668 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.905896 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.905983 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.908471 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:39.908476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:39.908478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:39.908521 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:39.909901 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.910349 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.910353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.910355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.910431 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.910743 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.910745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.910746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.910969 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:39.911056 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:39.911323 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 16:39:39.911448 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:39:39.913329 32027 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.25.71.124:38021: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2b1cd9df519e432b883536905633b131"
member_type: NON_VOTER
last_known_addr {
host: "127.25.71.124"
port: 38021
}
attrs {
promote: true
}
}
dest_uuid: "f350c795eedf4b9a91b21283d5e0f6d9"
cas_config_opid_index: 5
I20260928 16:39:39.914201 32027 consensus_queue.cc:237] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: NON_VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: true } }
I20260928 16:39:39.915474 32593 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2b1cd9df519e432b883536905633b131 (127.25.71.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: NON_VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: true } } }
I20260928 16:39:39.917363 32593 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.25.71.124:38021
I20260928 16:39:39.917575 32593 sys_catalog.cc:455] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: NON_VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: true } } }
I20260928 16:39:39.917698 32593 sys_catalog.cc:458] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:39.918421 32724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:39.920327 32724 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 1 FOLLOWER]: Refusing update from remote peer f350c795eedf4b9a91b21283d5e0f6d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 16:39:39.921538 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.915726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:44555 (local address 127.0.0.1:46786)
0928 16:39:39.916679 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:39.916695 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:39.916782 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:39.917289 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:39.917292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:39.917311 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:39.917422 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.917430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.918495 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.918500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.919789 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.919803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.921006 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.921011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.921135 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.921196 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 16:39:39.921275 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 16:39:39.921382 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882}
I20260928 16:39:39.922041 32755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.916034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.25.71.125:44555)
0928 16:39:39.916515 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:39.916520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:39.916868 (+ 348us) server_negotiation.cc:407] Connection header received
0928 16:39:39.916932 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:39.916935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:39.917010 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:39.917128 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:39.917566 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.918355 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.919955 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.920625 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.920734 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.920935 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:39.921804 (+ 869us) server_negotiation.cc:299] Negotiation successful
0928 16:39:39.921911 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385}
I20260928 16:39:39.922341 32592 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: NON_VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:39.922578 32643 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 1 LEARNER]: Refusing update from remote peer f350c795eedf4b9a91b21283d5e0f6d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 16:39:39.923107 32593 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 16:39:39.925472 32643 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2b1cd9df519e432b883536905633b131 (127.25.71.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: NON_VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: true } } }
I20260928 16:39:39.926855 32597 master.cc:537] Master@127.25.71.124:38021 shutting down...
I20260928 16:39:39.928509 32724 raft_consensus.cc:858] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: NON_VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: true } }
W20260928 16:39:39.928661 32724 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: NON_VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: true } }
I20260928 16:39:39.930778 32672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f350c795eedf4b9a91b21283d5e0f6d9. Latest consensus state: current_term: 1 leader_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: NON_VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: true } } }
I20260928 16:39:39.930946 32672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [sys.catalog]: This master's current role is: LEARNER
I20260928 16:39:39.931059 32672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: NON_VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: true } } }
I20260928 16:39:39.931162 32672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [sys.catalog]: This master's current role is: LEARNER
I20260928 16:39:39.931384 32724 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 91facbd2cee44b1f814eb251c69daf0a (127.25.71.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true } } }
I20260928 16:39:39.933280 32593 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9: attempting to promote NON_VOTER 91facbd2cee44b1f814eb251c69daf0a to VOTER
I20260928 16:39:39.933851 32593 consensus_queue.cc:237] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: NON_VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: true } }
I20260928 16:39:39.934223 32757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f350c795eedf4b9a91b21283d5e0f6d9. Latest consensus state: current_term: 1 leader_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true } } }
I20260928 16:39:39.934333 32757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 16:39:39.934424 32757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true } } }
I20260928 16:39:39.934484 32757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 16:39:39.934549 32757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: NON_VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: true } } }
I20260928 16:39:39.934619 32757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 16:39:39.936606 32643 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 1 LEARNER]: Refusing update from remote peer f350c795eedf4b9a91b21283d5e0f6d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260928 16:39:39.936805 32013 consensus_peers.cc:487] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 -> Peer 2b1cd9df519e432b883536905633b131 (127.25.71.124:38021): Couldn't send request to peer 2b1cd9df519e432b883536905633b131. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 16:39:39.937027 32593 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 16:39:39.939206 32593 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 91facbd2cee44b1f814eb251c69daf0a (127.25.71.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: NON_VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: true } } }
I20260928 16:39:39.939734 311 mvcc.cc:205] Tried to move back new op lower bound from 7334353216825925632 to 7334353216620380160. Current Snapshot: MvccSnapshot[applied={T|T < 7334353216666927104}]
I20260928 16:39:39.941911 32643 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 91facbd2cee44b1f814eb251c69daf0a (127.25.71.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: NON_VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: true } } }
I20260928 16:39:39.943732 312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: NON_VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: true } } }
I20260928 16:39:39.943962 312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:39:39.944331 32592 sys_catalog.cc:455] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: NON_VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: true } } }
I20260928 16:39:39.944468 32592 sys_catalog.cc:458] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:39.944658 32593 sys_catalog.cc:455] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: NON_VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: true } } }
I20260928 16:39:39.944756 32593 sys_catalog.cc:458] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:39.948320 32597 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260928 16:39:39.948670 32597 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260928 16:39:39.948763 32597 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: stopping tablet replica
I20260928 16:39:39.965633 32597 master.cc:552] Master@127.25.71.124:38021 shutdown complete.
I20260928 16:39:39.965720 32597 master_runner.cc:305] Clearing existing system tablet
I20260928 16:39:39.965998 32597 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:39.966068 32597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:39.967504 32597 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 16:39:39.968429 322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:39.968667 32597 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:39.968797 32597 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 16:39:39.968837 32597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/wal
uuid: "2b1cd9df519e432b883536905633b131"
format_stamp: "Formatted at 2026-09-28 16:39:39 on dist-test-slave-nqgh"
server_key: "1c066cf26d3fbf2d4e00dc071194c371"
server_key_iv: "1775ed85161af0f2f2e91ef67120f564"
server_key_version: "encryptionkey@0"
I20260928 16:39:39.971086 32597 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:39.973603 32597 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 16:39:39.973757 32597 log.cc:1196] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 16:39:39.974066 32597 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: Deleting consensus metadata
I20260928 16:39:39.974246 32597 master_runner.cc:315] Copying system tablet from 127.25.71.126:33855
I20260928 16:39:39.975826 32597 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: tablet copy: Beginning tablet copy session from remote peer at address 127.25.71.126:33855
I20260928 16:39:39.986486 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.976684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.25.71.126:33855)
0928 16:39:39.976863 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:39.976869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:39.978983 (+ 2114us) server_negotiation.cc:407] Connection header received
0928 16:39:39.979227 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:39.979230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:39.979287 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:39.979369 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:39.979891 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.980488 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.981351 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.981599 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.981693 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.983161 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:39.983165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:39.983168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:39.983196 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:39.984732 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.985174 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.985178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.985180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.985263 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:39.985510 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:39.985514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:39.985515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:39.985781 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:39.985872 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:39.986172 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 16:39:39.986299 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 16:39:39.986487 328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:39.976566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.0.0.1:56316)
0928 16:39:39.978820 (+ 2254us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:39.978838 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:39.979111 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:39.979401 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:39.979405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:39.979658 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:39.979773 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.979779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.980597 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.980601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.981239 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:39.981247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:39.981762 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:39.981765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:39.981862 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:39.982479 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:39.982491 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:39.982997 (+ 506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:39.984854 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.984858 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.984861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.985082 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.985344 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:39.985347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:39.985350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:39.985433 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:39.985915 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:39.985918 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:39.986011 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 16:39:39.986099 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 16:39:39.986288 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2167,"thread_start_us":115,"threads_started":1}
I20260928 16:39:39.987495 32048 tablet_copy_service.cc:140] P f350c795eedf4b9a91b21283d5e0f6d9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2b1cd9df519e432b883536905633b131 ({username='oryx', principal='oryx/127.25.71.124@KRBTEST.COM'} at 127.0.0.1:56316)
I20260928 16:39:39.987605 32048 tablet_copy_service.cc:161] P f350c795eedf4b9a91b21283d5e0f6d9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2b1cd9df519e432b883536905633b131 at {username='oryx', principal='oryx/127.25.71.124@KRBTEST.COM'} at 127.0.0.1:56316: session id = 2b1cd9df519e432b883536905633b131-00000000000000000000000000000000
I20260928 16:39:39.988636 32048 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 16:39:39.990159 32597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 16:39:39.992615 32597 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: tablet copy: Starting download of 0 data blocks...
I20260928 16:39:39.992831 32597 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: tablet copy: Starting download of 1 WAL segments...
I20260928 16:39:39.994463 32597 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 16:39:39.996666 32048 tablet_copy_service.cc:342] P f350c795eedf4b9a91b21283d5e0f6d9: Request end of tablet copy session 2b1cd9df519e432b883536905633b131-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.25.71.124@KRBTEST.COM'} at 127.0.0.1:56316
I20260928 16:39:39.996766 32048 tablet_copy_service.cc:433] P f350c795eedf4b9a91b21283d5e0f6d9: ending tablet copy session 2b1cd9df519e432b883536905633b131-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2b1cd9df519e432b883536905633b131
I20260928 16:39:40.005754 32597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:40.007917 329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:40.007917 330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:40.014439 32597 server_base.cc:1034] running on GCE node
W20260928 16:39:40.014880 332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:40.015257 32597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:40.015731 32597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:40.016883 32597 hybrid_clock.cc:648] HybridClock initialized: now 1790613580016872 us; error 41 us; skew 500 ppm
Sep 28 16:39:40 dist-test-slave-nqgh krb5kdc[31983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613580, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:40.018723 32597 init.cc:377] Logged in from keytab as oryx/127.25.71.124@KRBTEST.COM (short username oryx)
I20260928 16:39:40.019309 32597 webserver.cc:460] Webserver started at http://127.25.71.124:33491/ using document root <none> and password file <none>
I20260928 16:39:40.019515 32597 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:40.019554 32597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:40.020932 32597 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 16:39:40.022039 339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:40.022301 32597 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 16:39:40.022380 32597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/wal
uuid: "2b1cd9df519e432b883536905633b131"
format_stamp: "Formatted at 2026-09-28 16:39:39 on dist-test-slave-nqgh"
server_key: "1c066cf26d3fbf2d4e00dc071194c371"
server_key_iv: "1775ed85161af0f2f2e91ef67120f564"
server_key_version: "encryptionkey@0"
I20260928 16:39:40.022488 32597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:40.052616 32597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:40.054826 32597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:40.060031 32597 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.124:38021
I20260928 16:39:40.060038 394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.124:38021 every 8 connection(s)
I20260928 16:39:40.060722 32597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/master-2/data/info.pb
I20260928 16:39:40.062888 396 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:39:40.063975 396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: Bootstrap starting.
W20260928 16:39:40.064566 309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:40.065018 309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:40.065110 309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:40.071025 309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:40.071116 309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:40.071180 309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:39:40.072465 396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:39:40.072781 396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: Bootstrap complete.
I20260928 16:39:40.073364 396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: NON_VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: true } }
I20260928 16:39:40.073508 396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2b1cd9df519e432b883536905633b131, State: Initialized, Role: LEARNER
I20260928 16:39:40.073621 396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: NON_VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: true } }
I20260928 16:39:40.073989 399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: NON_VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: true } } }
I20260928 16:39:40.074121 399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [sys.catalog]: This master's current role is: LEARNER
I20260928 16:39:40.074185 396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:40.078009 309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.65.keytab
--principal=oryx/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:43829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=37095
--tserver_master_addrs=127.25.71.126:33855,127.25.71.125:44555,127.25.71.124:38021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:40.078696 410 catalog_manager.cc:1259] Loaded cluster ID: 521ae1f785a849e1a4661590fe3180c4
I20260928 16:39:40.078773 410 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: loading cluster ID for follower catalog manager: success
I20260928 16:39:40.079571 309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:40.080782 309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:39:40.080847 410 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: acquiring CA information for follower catalog manager: success
I20260928 16:39:40.082422 410 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 16:39:40.092911 412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:40.092957 415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:40.092962 413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:40.093327 309 server_base.cc:1034] running on GCE node
I20260928 16:39:40.094002 309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:40.094959 309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:40.096257 309 hybrid_clock.cc:648] HybridClock initialized: now 1790613580096235 us; error 39 us; skew 500 ppm
Sep 28 16:39:40 dist-test-slave-nqgh krb5kdc[31983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613580, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:40.100137 309 init.cc:377] Logged in from keytab as oryx/127.25.71.65@KRBTEST.COM (short username oryx)
I20260928 16:39:40.101480 309 webserver.cc:460] Webserver started at http://127.25.71.65:37095/ using document root <none> and password file <none>
I20260928 16:39:40.102253 309 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:40.102357 309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:40.108054 309 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:40.111265 422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:40.113364 309 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:40.113612 309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "a53637373b524375834e5110de5e7587"
format_stamp: "Formatted at 2026-09-28 16:39:38 on dist-test-slave-nqgh"
server_key: "945eacf6a85b09d920715a1a1ec9dfe6"
server_key_iv: "f21301591516f64df03ea4324bfa027f"
server_key_version: "encryptionkey@0"
I20260928 16:39:40.114557 309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:40.140540 32491 heartbeater.cc:503] Master 127.25.71.126:33855 was elected leader, sending a full tablet report...
I20260928 16:39:40.154767 309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:40.158875 309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:40.159086 309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:40.159812 309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:40.160939 309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:40.160992 309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:40.161039 309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:40.161068 309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:40.172957 309 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:43829
I20260928 16:39:40.172977 537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:43829 every 8 connection(s)
I20260928 16:39:40.174518 309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:39:40.177791 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 309
I20260928 16:39:40.178021 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 32226
I20260928 16:39:40.186898 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:36905
--local_ip_for_outbound_sockets=127.25.71.66
--tserver_master_addrs=127.25.71.126:33855
--webserver_port=42299
--webserver_interface=127.25.71.66
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.66.keytab
--principal=oryx/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.25.71.126:33855,127.25.71.125:44555
--tserver_master_addrs=127.25.71.126:33855,127.25.71.125:44555,127.25.71.124:38021 with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:39:40 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613580, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.124@KRBTEST.COM
Sep 28 16:39:40 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613580, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.125@KRBTEST.COM
Sep 28 16:39:40 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613580, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM
I20260928 16:39:40.198411 32755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.177495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:56545 (local address 127.25.71.125:44555)
0928 16:39:40.177683 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:40.177688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:40.179919 (+ 2231us) server_negotiation.cc:407] Connection header received
0928 16:39:40.181376 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:40.181381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:40.181457 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:40.181514 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:40.184359 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.185261 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.187537 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.187912 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.187995 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.192760 (+ 4765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:40.192767 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:40.192770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:40.192806 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:40.195184 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.196073 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.196077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.196079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.196182 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.196736 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.196739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.196741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.197008 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:40.197092 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:40.198075 (+ 983us) server_negotiation.cc:299] Negotiation successful
0928 16:39:40.198234 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 16:39:40.198423 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.180456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:59009 (local address 127.25.71.126:33855)
0928 16:39:40.180602 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:40.180606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:40.182523 (+ 1917us) server_negotiation.cc:407] Connection header received
0928 16:39:40.182824 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:40.182827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:40.182890 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:40.182944 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:40.184281 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.185115 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.187530 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.187912 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.187995 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.193171 (+ 5176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:40.193176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:40.193179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:40.193213 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:40.195184 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.196073 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.196077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.196079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.196289 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.196736 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.196739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.196741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.197008 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:40.197092 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:40.198084 (+ 992us) server_negotiation.cc:299] Negotiation successful
0928 16:39:40.198234 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 16:39:40.199913 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.177502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:34329 (local address 127.25.71.124:38021)
0928 16:39:40.177832 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:40.177837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:40.181261 (+ 3424us) server_negotiation.cc:407] Connection header received
0928 16:39:40.181390 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:40.181394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:40.181467 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:40.181551 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:40.184359 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.185466 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.187856 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.188220 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.188308 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.193270 (+ 4962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:40.193275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:40.193279 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:40.193321 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:40.195046 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.196110 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.196114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.196116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.196236 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.196758 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.196761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.196763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.197072 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:40.197177 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:40.199498 (+ 2321us) server_negotiation.cc:299] Negotiation successful
0928 16:39:40.199692 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":101,"threads_started":1}
I20260928 16:39:40.200358 544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.179963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.65:59009)
0928 16:39:40.182413 (+ 2450us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:40.182429 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:40.182663 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:40.183126 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:40.183129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:40.183379 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:40.184083 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.184111 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.185328 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.185331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.187353 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.187369 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.188237 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.188241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.188593 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.189517 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:40.189530 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:40.192949 (+ 3419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:40.195342 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:40.195346 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:40.195349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:40.195937 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:40.196387 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:40.196390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:40.196392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:40.196619 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:40.197166 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:40.197174 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:40.197718 (+ 544us) client_negotiation.cc:769] Sending connection context
0928 16:39:40.199077 (+ 1359us) client_negotiation.cc:240] Negotiation successful
0928 16:39:40.199215 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2352,"mutex_wait_us":304,"spinlock_wait_cycles":21248,"thread_start_us":105,"threads_started":1}
I20260928 16:39:40.200647 545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.178162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:38021 (local address 127.25.71.65:34329)
0928 16:39:40.181100 (+ 2938us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:40.181116 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:40.181276 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:40.181836 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:40.181852 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:40.182804 (+ 952us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:40.184086 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.184116 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.185605 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.185608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.187667 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.187681 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.188476 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.188479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.188741 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.189628 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:40.189647 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:40.192978 (+ 3331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:40.195260 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:40.195271 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:40.195302 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:40.195962 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:40.196355 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:40.196359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:40.196361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:40.196631 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:40.197240 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:40.197243 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:40.198051 (+ 808us) client_negotiation.cc:769] Sending connection context
0928 16:39:40.198213 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 16:39:40.198800 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2826,"mutex_wait_us":769,"spinlock_wait_cycles":11648,"thread_start_us":788,"threads_started":1}
I20260928 16:39:40.200878 543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.178161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:44555 (local address 127.25.71.65:56545)
0928 16:39:40.179691 (+ 1530us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:40.179774 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:40.181056 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:40.181839 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:40.181852 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:40.182804 (+ 952us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:40.184086 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.184116 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.185438 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.185442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.187388 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.187395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.188237 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.188241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.188621 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.189459 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:40.189484 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:40.192468 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:40.195342 (+ 2874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:40.195346 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:40.195349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:40.195937 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:40.196392 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:40.196395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:40.196396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:40.196619 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:40.197166 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:40.197174 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:40.197718 (+ 544us) client_negotiation.cc:769] Sending connection context
0928 16:39:40.198684 (+ 966us) client_negotiation.cc:240] Negotiation successful
0928 16:39:40.198906 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1257,"mutex_wait_us":552,"spinlock_wait_cycles":48896,"thread_start_us":153,"threads_started":1}
I20260928 16:39:40.202148 540 heartbeater.cc:347] Connected to a master server at 127.25.71.125:44555
I20260928 16:39:40.202230 539 heartbeater.cc:347] Connected to a master server at 127.25.71.126:33855
I20260928 16:39:40.202345 538 heartbeater.cc:347] Connected to a master server at 127.25.71.124:38021
I20260928 16:39:40.202483 538 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:40.202443 540 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:40.202443 539 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:40.203387 538 heartbeater.cc:511] Master 127.25.71.124:38021 requested a full tablet report, sending...
I20260928 16:39:40.203387 539 heartbeater.cc:511] Master 127.25.71.126:33855 requested a full tablet report, sending...
I20260928 16:39:40.203387 540 heartbeater.cc:511] Master 127.25.71.125:44555 requested a full tablet report, sending...
I20260928 16:39:40.205556 32027 ts_manager.cc:194] Re-registered known tserver with Master: a53637373b524375834e5110de5e7587 (127.25.71.65:43829)
I20260928 16:39:40.205575 32633 ts_manager.cc:194] Registered new tserver with Master: a53637373b524375834e5110de5e7587 (127.25.71.65:43829)
I20260928 16:39:40.205725 359 ts_manager.cc:194] Registered new tserver with Master: a53637373b524375834e5110de5e7587 (127.25.71.65:43829)
I20260928 16:39:40.206600 32027 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:59009
W20260928 16:39:40.331791 546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:40.332180 546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:40.332231 546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:40.336829 546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:40.336932 546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:40.337002 546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:39:40.343447 546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.66.keytab
--principal=oryx/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:36905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=42299
--tserver_master_addrs=127.25.71.126:33855,127.25.71.125:44555,127.25.71.124:38021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:40.345019 546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:40.346406 546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:40.358973 552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:40.358986 551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:40.358995 554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:40.359048 546 server_base.cc:1034] running on GCE node
I20260928 16:39:40.359860 546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:40.360606 546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:40.361820 546 hybrid_clock.cc:648] HybridClock initialized: now 1790613580361785 us; error 48 us; skew 500 ppm
Sep 28 16:39:40 dist-test-slave-nqgh krb5kdc[31983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613580, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:40.365202 546 init.cc:377] Logged in from keytab as oryx/127.25.71.66@KRBTEST.COM (short username oryx)
I20260928 16:39:40.366464 546 webserver.cc:460] Webserver started at http://127.25.71.66:42299/ using document root <none> and password file <none>
I20260928 16:39:40.367210 546 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:40.367326 546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:40.372272 546 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:39:40.375300 561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:40.377420 546 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:40.377697 546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "b0d7f25c092a41cc93edb7430142e0db"
format_stamp: "Formatted at 2026-09-28 16:39:38 on dist-test-slave-nqgh"
server_key: "c17f4125468c818e54bcb779611ade0d"
server_key_iv: "f4bb66c221cbd4f0db6c0875177b13b4"
server_key_version: "encryptionkey@0"
I20260928 16:39:40.378465 546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:40.401719 546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:40.405762 546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:40.405987 546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:40.406699 546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:40.407783 546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:40.407840 546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:40.407899 546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:40.407929 546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:40.419042 546 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:36905
I20260928 16:39:40.419077 676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:36905 every 8 connection(s)
I20260928 16:39:40.420353 546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:39:40.427618 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 546
I20260928 16:39:40.427829 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 32361
I20260928 16:39:40.433351 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.425893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:38021 (local address 127.0.0.1:55370)
0928 16:39:40.426089 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:40.426103 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:40.426162 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:40.427150 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:40.427153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:40.427176 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:40.427452 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.427468 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.428745 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.428749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.429970 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.429985 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.432823 (+ 2838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.432827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.432935 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.432968 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 16:39:40.433024 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 16:39:40.433147 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":3}
Sep 28 16:39:40 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613580, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.124@KRBTEST.COM
I20260928 16:39:40.435977 684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.425940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.25.71.124:38021)
0928 16:39:40.426841 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:40.426846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:40.426859 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:39:40.426899 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:40.426902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:40.426965 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:40.427654 (+ 689us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:40.427771 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.428575 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.430864 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.431569 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.433973 (+ 2404us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.434217 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:40.434268 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 16:39:40.435758 (+ 1490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":81,"threads_started":1}
Sep 28 16:39:40 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613580, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.125@KRBTEST.COM
I20260928 16:39:40.436779 369 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 16:39:40.437125 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:44409
--local_ip_for_outbound_sockets=127.25.71.67
--tserver_master_addrs=127.25.71.126:33855
--webserver_port=37883
--webserver_interface=127.25.71.67
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.67.keytab
--principal=oryx/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.25.71.126:33855,127.25.71.125:44555
--tserver_master_addrs=127.25.71.126:33855,127.25.71.125:44555,127.25.71.124:38021 with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:39:40.437161 399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f350c795eedf4b9a91b21283d5e0f6d9. Latest consensus state: current_term: 1 leader_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: NON_VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: true } } }
I20260928 16:39:40.437358 399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [sys.catalog]: This master's current role is: LEARNER
Sep 28 16:39:40 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613580, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM
I20260928 16:39:40.443202 32755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.422790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:37371 (local address 127.25.71.125:44555)
0928 16:39:40.422945 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:40.422949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:40.424758 (+ 1809us) server_negotiation.cc:407] Connection header received
0928 16:39:40.425870 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:40.425874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:40.425937 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:40.425993 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:40.431184 (+ 5191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.432015 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.433295 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.433653 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.433735 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.438028 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:40.438033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:40.438037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:40.438077 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:40.439853 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.440791 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.440795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.440797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.440893 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.441374 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.441377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.441379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.441667 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:40.441746 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:40.442813 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0928 16:39:40.442991 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 16:39:40.443202 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.425043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:51581 (local address 127.25.71.124:38021)
0928 16:39:40.425207 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:40.425213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:40.425240 (+ 27us) server_negotiation.cc:407] Connection header received
0928 16:39:40.426389 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:40.426392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:40.426454 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:40.426506 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:40.428805 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.429591 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.431930 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.432338 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.432420 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.438404 (+ 5984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:40.438408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:40.438412 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:40.438444 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:40.439949 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.440791 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.440795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.440797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.440894 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.441374 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.441377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.441379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.441659 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:40.441746 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:40.442813 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0928 16:39:40.442992 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:39:40.445536 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.423018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:42159 (local address 127.25.71.126:33855)
0928 16:39:40.423198 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:40.423203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:40.424451 (+ 1248us) server_negotiation.cc:407] Connection header received
0928 16:39:40.425882 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:40.425886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:40.425951 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:40.426007 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:40.428530 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.430676 (+ 2146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.434671 (+ 3995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.435016 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.435102 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.440703 (+ 5601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:40.440707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:40.440710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:40.440747 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:40.442757 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.443925 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.443929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.443931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.444040 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.444376 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.444380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.444381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.444662 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:40.444741 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:40.445154 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 16:39:40.445322 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1283,"server-negotiator.queue_time_us":73}
I20260928 16:39:40.445523 683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.424585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:38021 (local address 127.25.71.66:51581)
0928 16:39:40.424925 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:40.424941 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:40.426245 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:40.426702 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:40.426704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:40.427400 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:40.428657 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.428672 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.429813 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.429817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.431799 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.431808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.432720 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.432725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.432995 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.434004 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:40.434042 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:40.437311 (+ 3269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:40.440177 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:40.440180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:40.440182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:40.440663 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:40.440999 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:40.441002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:40.441004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:40.441258 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:40.441824 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:40.441834 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:40.442398 (+ 564us) client_negotiation.cc:769] Sending connection context
0928 16:39:40.443522 (+ 1124us) client_negotiation.cc:240] Negotiation successful
0928 16:39:40.444069 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":201,"spinlock_wait_cycles":21504,"thread_start_us":126,"threads_started":1}
I20260928 16:39:40.445883 682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.423395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:44555 (local address 127.25.71.66:37371)
0928 16:39:40.424652 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:40.424669 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:40.425592 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:40.426488 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:40.426497 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:40.427289 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:40.430985 (+ 3696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.430996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.432182 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.432186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.433143 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.433155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.433931 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.433934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.434047 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.434617 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:40.434662 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:40.437819 (+ 3157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:40.440034 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:40.440044 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:40.440075 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:40.440663 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:40.440999 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:40.441002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:40.441004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:40.441258 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:40.441824 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:40.441834 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:40.442398 (+ 564us) client_negotiation.cc:769] Sending connection context
0928 16:39:40.443447 (+ 1049us) client_negotiation.cc:240] Negotiation successful
0928 16:39:40.443996 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1146,"mutex_wait_us":160,"spinlock_wait_cycles":39936,"thread_start_us":810,"threads_started":1}
I20260928 16:39:40.446769 681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.423394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.66:42159)
0928 16:39:40.424181 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:40.424272 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:40.425579 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:40.426484 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:40.426498 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:40.427289 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:40.428250 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.428277 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.431712 (+ 3435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.431717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.432639 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.432647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.435288 (+ 2641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.435292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.435404 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.436688 (+ 1284us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:40.436706 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:40.440475 (+ 3769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:40.442976 (+ 2501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:40.442980 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:40.442983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:40.443775 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:40.444168 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:40.444171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:40.444173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:40.444268 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:40.444817 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:40.444820 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:40.444951 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 16:39:40.446290 (+ 1339us) client_negotiation.cc:240] Negotiation successful
0928 16:39:40.446519 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":2,"spinlock_wait_cycles":24320,"thread_start_us":181,"threads_started":1}
I20260928 16:39:40.447441 677 heartbeater.cc:347] Connected to a master server at 127.25.71.124:38021
I20260928 16:39:40.447454 679 heartbeater.cc:347] Connected to a master server at 127.25.71.125:44555
I20260928 16:39:40.447803 679 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:40.447806 677 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:40.448036 678 heartbeater.cc:347] Connected to a master server at 127.25.71.126:33855
I20260928 16:39:40.448132 678 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:40.448671 679 heartbeater.cc:511] Master 127.25.71.125:44555 requested a full tablet report, sending...
I20260928 16:39:40.448674 677 heartbeater.cc:511] Master 127.25.71.124:38021 requested a full tablet report, sending...
I20260928 16:39:40.448679 678 heartbeater.cc:511] Master 127.25.71.126:33855 requested a full tablet report, sending...
I20260928 16:39:40.450176 32027 ts_manager.cc:194] Re-registered known tserver with Master: b0d7f25c092a41cc93edb7430142e0db (127.25.71.66:36905)
I20260928 16:39:40.450261 32633 ts_manager.cc:194] Registered new tserver with Master: b0d7f25c092a41cc93edb7430142e0db (127.25.71.66:36905)
I20260928 16:39:40.450261 359 ts_manager.cc:194] Registered new tserver with Master: b0d7f25c092a41cc93edb7430142e0db (127.25.71.66:36905)
I20260928 16:39:40.451094 32027 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:42159
W20260928 16:39:40.578960 685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:40.579286 685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:40.579357 685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:40.583806 685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:40.583892 685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:40.583978 685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:39:40.589833 685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.67.keytab
--principal=oryx/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:44409
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=37883
--tserver_master_addrs=127.25.71.126:33855,127.25.71.125:44555,127.25.71.124:38021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:40.591070 685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:40.592146 685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:40.603438 690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:40.603493 691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:40.603905 693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:40.604146 685 server_base.cc:1034] running on GCE node
I20260928 16:39:40.604709 685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:40.605573 685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:40.606765 685 hybrid_clock.cc:648] HybridClock initialized: now 1790613580606737 us; error 49 us; skew 500 ppm
Sep 28 16:39:40 dist-test-slave-nqgh krb5kdc[31983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613580, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:40.610100 685 init.cc:377] Logged in from keytab as oryx/127.25.71.67@KRBTEST.COM (short username oryx)
I20260928 16:39:40.611290 685 webserver.cc:460] Webserver started at http://127.25.71.67:37883/ using document root <none> and password file <none>
I20260928 16:39:40.611889 685 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:40.611995 685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:40.616947 685 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:39:40.620066 700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:40.622218 685 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:40.622401 685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "83b2286be80340039d380a6bacdb8e43"
format_stamp: "Formatted at 2026-09-28 16:39:39 on dist-test-slave-nqgh"
server_key: "405a454c3ae59014a4b93b54e2e980f3"
server_key_iv: "db453fb7839975159b5f00d5e78fd2fb"
server_key_version: "encryptionkey@0"
I20260928 16:39:40.623246 685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:40.643692 685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:40.647099 685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:40.647292 685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:40.647930 685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:40.648846 685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:40.648898 685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:40.648954 685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:40.648981 685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:40.659823 685 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:44409
I20260928 16:39:40.659873 814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:44409 every 8 connection(s)
I20260928 16:39:40.660979 685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:39:40.667137 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 685
Sep 28 16:39:40 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613580, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM
Sep 28 16:39:40 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613580, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.125@KRBTEST.COM
Sep 28 16:39:40 dist-test-slave-nqgh krb5kdc[31983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613580, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.124@KRBTEST.COM
I20260928 16:39:40.682530 684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.670260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55386 (local address 127.25.71.124:38021)
0928 16:39:40.670403 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:40.670407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:40.672005 (+ 1598us) server_negotiation.cc:407] Connection header received
0928 16:39:40.672216 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:40.672219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:40.672271 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:40.672325 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:40.673389 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.674118 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.675448 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.675771 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.675947 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.677845 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:40.677849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:40.677852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:40.677886 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:40.679960 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.680871 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.680876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.680878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.680990 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.681391 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.681394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.681396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.681629 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:40.681764 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:40.682173 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 16:39:40.682348 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:39:40.683091 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.663354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:59937 (local address 127.25.71.126:33855)
0928 16:39:40.663539 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:40.663545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:40.664824 (+ 1279us) server_negotiation.cc:407] Connection header received
0928 16:39:40.666324 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:40.666328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:40.666399 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:40.666482 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:40.668973 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.669839 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.672018 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.672355 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.672447 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.677226 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:40.677230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:40.677233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:40.677264 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:40.679269 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.680103 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.680107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.680109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.680358 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.680940 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.680943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.680945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.681225 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:40.681322 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:40.682783 (+ 1461us) server_negotiation.cc:299] Negotiation successful
0928 16:39:40.682943 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:39:40.684518 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.663887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.67:59937)
0928 16:39:40.664581 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:40.664674 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:40.666013 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:40.666830 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:40.666841 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:40.667690 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:40.668730 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.668760 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.670045 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.670048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.671843 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.671859 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.672656 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.672660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.672904 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.673683 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:40.673706 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:40.676986 (+ 3280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:40.679489 (+ 2503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:40.679497 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:40.679523 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:40.679961 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:40.680476 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:40.680479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:40.680482 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:40.680725 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:40.681397 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:40.681406 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:40.681852 (+ 446us) client_negotiation.cc:769] Sending connection context
0928 16:39:40.682093 (+ 241us) client_negotiation.cc:240] Negotiation successful
0928 16:39:40.683248 (+ 1155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":83,"spinlock_wait_cycles":36864,"thread_start_us":707,"threads_started":1}
I20260928 16:39:40.684787 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.665392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:42299 (local address 127.25.71.124:38021)
0928 16:39:40.665531 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:40.665536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:40.665658 (+ 122us) server_negotiation.cc:407] Connection header received
0928 16:39:40.666322 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:40.666326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:40.666391 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:40.666446 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:40.668971 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.669831 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.675271 (+ 5440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.675638 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.675723 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.678532 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:40.678536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:40.678538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:40.678562 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:40.680629 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.682606 (+ 1977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.682610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.682612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.682731 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.683197 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.683200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.683202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.683463 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:40.683534 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:40.684456 (+ 922us) server_negotiation.cc:299] Negotiation successful
0928 16:39:40.684605 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:39:40.685429 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.669799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.25.71.126:33855)
0928 16:39:40.670081 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:40.670086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:40.670929 (+ 843us) server_negotiation.cc:407] Connection header received
0928 16:39:40.670986 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:40.670989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:40.671057 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:40.671156 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:40.672284 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.673036 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.674208 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.674563 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.674686 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.679867 (+ 5181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:40.679871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:40.679874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:40.679902 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:40.681632 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.682171 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.682174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.682175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.682271 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.682564 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.682567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.682569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.683875 (+ 1306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:40.684014 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:40.685084 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0928 16:39:40.685258 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
I20260928 16:39:40.685652 821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.665265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:38021 (local address 127.25.71.67:42299)
0928 16:39:40.665555 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:40.665569 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:40.666047 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:40.666827 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:40.666839 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:40.667684 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:40.668733 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.668760 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.670045 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.670048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.673443 (+ 3395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.673453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.676121 (+ 2668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.676125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.676234 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.676847 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:40.676863 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:40.678375 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:40.682164 (+ 3789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:40.682168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:40.682170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:40.682433 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:40.682967 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:40.682970 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:40.682972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:40.683060 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:40.683761 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:40.683764 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:40.683870 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 16:39:40.683965 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 16:39:40.684175 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":1910,"spinlock_wait_cycles":17536,"thread_start_us":104,"threads_started":1}
I20260928 16:39:40.686498 816 heartbeater.cc:347] Connected to a master server at 127.25.71.126:33855
I20260928 16:39:40.686498 815 heartbeater.cc:347] Connected to a master server at 127.25.71.124:38021
I20260928 16:39:40.686934 816 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:40.686935 815 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:40.687659 816 heartbeater.cc:511] Master 127.25.71.126:33855 requested a full tablet report, sending...
I20260928 16:39:40.687880 815 heartbeater.cc:511] Master 127.25.71.124:38021 requested a full tablet report, sending...
I20260928 16:39:40.689194 32028 ts_manager.cc:194] Re-registered known tserver with Master: 83b2286be80340039d380a6bacdb8e43 (127.25.71.67:44409)
I20260928 16:39:40.689302 359 ts_manager.cc:194] Registered new tserver with Master: 83b2286be80340039d380a6bacdb8e43 (127.25.71.67:44409)
I20260928 16:39:40.689958 32028 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:59937
I20260928 16:39:40.690860 32755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.663454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:52819 (local address 127.25.71.125:44555)
0928 16:39:40.663594 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:40.663599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:40.664984 (+ 1385us) server_negotiation.cc:407] Connection header received
0928 16:39:40.666663 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:40.666667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:40.666724 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:40.666807 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:40.670533 (+ 3726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.671435 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.672728 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.673077 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.673153 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.678383 (+ 5230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:40.678388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:40.678391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:40.678427 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:40.681964 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.686257 (+ 4293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.686263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.686265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.686395 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.688849 (+ 2454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.688854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.688856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.689125 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:40.689211 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:40.690447 (+ 1236us) server_negotiation.cc:299] Negotiation successful
0928 16:39:40.690670 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:39:40.691553 820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.663887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:44555 (local address 127.25.71.67:52819)
0928 16:39:40.664890 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:40.664904 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:40.666013 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:40.666929 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:40.666932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:40.667684 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:40.668733 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.668760 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.671590 (+ 2830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.671593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.672593 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.672604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.673701 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.673705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.673862 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.674442 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:40.674458 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:40.677481 (+ 3023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:40.685819 (+ 8338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:40.685822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:40.685825 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:40.686093 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:40.688569 (+ 2476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:40.688573 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:40.688576 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:40.688713 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:40.689570 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:40.689575 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:40.689689 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 16:39:40.691123 (+ 1434us) client_negotiation.cc:240] Negotiation successful
0928 16:39:40.691338 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"mutex_wait_us":152,"spinlock_wait_cycles":97152,"thread_start_us":183,"threads_started":1}
I20260928 16:39:40.692658 817 heartbeater.cc:347] Connected to a master server at 127.25.71.125:44555
I20260928 16:39:40.692788 817 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:40.693128 817 heartbeater.cc:511] Master 127.25.71.125:44555 requested a full tablet report, sending...
I20260928 16:39:40.690752 32027 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 16:39:40.694077 32633 ts_manager.cc:194] Registered new tserver with Master: 83b2286be80340039d380a6bacdb8e43 (127.25.71.67:44409)
W20260928 16:39:40.695134 32027 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:39:40.697036 829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.676834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46798 (local address 127.25.71.125:44555)
0928 16:39:40.686803 (+ 9969us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:40.686808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:40.686822 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:39:40.686861 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:40.686864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:40.686917 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:40.687795 (+ 878us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:40.688392 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.689660 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.690915 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.691256 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.691354 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.693372 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:40.693377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:40.693380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:40.693421 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:40.695280 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.695755 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.695760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.695762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.695849 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:40.696106 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:40.696109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:40.696111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:40.696337 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:40.696463 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:40.696697 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 16:39:40.696838 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9764,"thread_start_us":108,"threads_started":1}
I20260928 16:39:40.698180 316 sys_catalog.cc:455] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: NON_VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: true } } }
I20260928 16:39:40.698355 316 sys_catalog.cc:458] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:40.702351 833 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9: attempting to promote NON_VOTER 2b1cd9df519e432b883536905633b131 to VOTER
I20260928 16:39:40.702854 833 consensus_queue.cc:237] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: false } }
I20260928 16:39:40.704447 32643 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 1 FOLLOWER]: Refusing update from remote peer f350c795eedf4b9a91b21283d5e0f6d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 16:39:40.705240 316 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 16:39:40.706390 369 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 1 LEARNER]: Refusing update from remote peer f350c795eedf4b9a91b21283d5e0f6d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 16:39:40.706890 316 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 16:39:40.707782 832 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2b1cd9df519e432b883536905633b131 (127.25.71.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: false } } }
I20260928 16:39:40.708320 32643 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2b1cd9df519e432b883536905633b131 (127.25.71.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: false } } }
I20260928 16:39:40.709321 316 sys_catalog.cc:455] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: false } } }
I20260928 16:39:40.709448 316 sys_catalog.cc:458] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:40.709333 832 sys_catalog.cc:455] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: false } } }
I20260928 16:39:40.709648 832 sys_catalog.cc:458] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:40.710769 316 sys_catalog.cc:455] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: false } } }
I20260928 16:39:40.710891 316 sys_catalog.cc:458] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:40.711138 369 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2b1cd9df519e432b883536905633b131 (127.25.71.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: false } } }
I20260928 16:39:40.711383 841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: false } } }
I20260928 16:39:40.711549 841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:39:40.715905 399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: false } } }
I20260928 16:39:40.716122 399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:39:40.720141 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.713402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:43829 (local address 127.0.0.1:57590)
0928 16:39:40.714123 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:40.714140 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:40.714245 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:40.715073 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:40.715077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:40.715100 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:40.715266 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.715274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.717594 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.717598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.718408 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.718418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.719697 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.719701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.719803 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.719837 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 16:39:40.719883 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 16:39:40.719966 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635}
I20260928 16:39:40.720850 843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.714216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:44409 (local address 127.0.0.1:50806)
0928 16:39:40.714585 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:40.714600 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:40.714728 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:40.716086 (+ 1358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:40.716089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:40.716110 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:40.716243 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.716250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.718039 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.718043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.719204 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.719218 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.720369 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.720373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.720487 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.720520 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 16:39:40.720571 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 16:39:40.720665 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":1,"thread_start_us":187,"threads_started":1}
I20260928 16:39:40.720991 842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.713757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57590 (local address 127.25.71.65:43829)
0928 16:39:40.714330 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:40.714344 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:40.714388 (+ 44us) server_negotiation.cc:407] Connection header received
0928 16:39:40.714507 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:40.714515 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:40.714881 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:40.716442 (+ 1561us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:40.716626 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.717469 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.718554 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.719554 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.720484 (+ 930us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.720667 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:40.720734 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 16:39:40.720841 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":91,"threads_started":1}
I20260928 16:39:40.721514 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.714541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:36905 (local address 127.0.0.1:39826)
0928 16:39:40.714886 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:40.714905 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:40.714986 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:40.716906 (+ 1920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:40.716909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:40.716929 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:40.717021 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.717026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.718516 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.718519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.719701 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.719710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.721114 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.721117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.721222 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.721244 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 16:39:40.721293 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 16:39:40.721372 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":148,"threads_started":1}
I20260928 16:39:40.721705 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.715067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50806 (local address 127.25.71.67:44409)
0928 16:39:40.715627 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:40.715637 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:40.715657 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:39:40.715723 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:40.715727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:40.715899 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:40.716419 (+ 520us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:40.716750 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.717890 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.719363 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.720227 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.721204 (+ 977us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.721388 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:40.721445 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 16:39:40.721553 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":173,"threads_started":1}
I20260928 16:39:40.722242 845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.715105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39826 (local address 127.25.71.66:36905)
0928 16:39:40.715446 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:40.715460 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:40.715485 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:39:40.715575 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:40.715581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:40.716747 (+ 1166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:40.717180 (+ 433us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:40.717313 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.718373 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.720114 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.720960 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.721809 (+ 849us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.721961 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:40.722012 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 16:39:40.722106 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":129,"threads_started":1}
I20260928 16:39:40.724925 470 tablet_service.cc:1467] Processing CreateTablet for tablet 7207dec97cc04f72ae8cd118cc48d580 (DEFAULT_TABLE table=test-table [id=3283141649d04b7fa5fe58cbb6d80421]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:40.725232 748 tablet_service.cc:1467] Processing CreateTablet for tablet 7207dec97cc04f72ae8cd118cc48d580 (DEFAULT_TABLE table=test-table [id=3283141649d04b7fa5fe58cbb6d80421]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:40.726236 470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7207dec97cc04f72ae8cd118cc48d580. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:40.726374 748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7207dec97cc04f72ae8cd118cc48d580. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:40.729590 610 tablet_service.cc:1467] Processing CreateTablet for tablet 7207dec97cc04f72ae8cd118cc48d580 (DEFAULT_TABLE table=test-table [id=3283141649d04b7fa5fe58cbb6d80421]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:40.731098 610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7207dec97cc04f72ae8cd118cc48d580. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:40.735366 847 tablet_bootstrap.cc:492] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43: Bootstrap starting.
I20260928 16:39:40.736017 848 tablet_bootstrap.cc:492] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587: Bootstrap starting.
I20260928 16:39:40.737943 847 tablet_bootstrap.cc:654] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:40.738101 848 tablet_bootstrap.cc:654] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:40.738957 847 log.cc:824] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:40.738957 848 log.cc:824] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:40.741359 847 tablet_bootstrap.cc:492] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43: No bootstrap required, opened a new log
I20260928 16:39:40.741359 848 tablet_bootstrap.cc:492] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587: No bootstrap required, opened a new log
I20260928 16:39:40.741379 851 tablet_bootstrap.cc:492] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db: Bootstrap starting.
I20260928 16:39:40.741595 847 ts_tablet_manager.cc:1397] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260928 16:39:40.741595 848 ts_tablet_manager.cc:1397] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:39:40.744516 851 tablet_bootstrap.cc:654] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:40.745560 851 log.cc:824] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:40.745805 848 raft_consensus.cc:348] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53637373b524375834e5110de5e7587" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 43829 } } peers { permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 } } peers { permanent_uuid: "83b2286be80340039d380a6bacdb8e43" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 44409 } }
I20260928 16:39:40.746202 848 raft_consensus.cc:374] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:40.746271 848 raft_consensus.cc:729] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a53637373b524375834e5110de5e7587, State: Initialized, Role: FOLLOWER
I20260928 16:39:40.746866 848 consensus_queue.cc:260] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53637373b524375834e5110de5e7587" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 43829 } } peers { permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 } } peers { permanent_uuid: "83b2286be80340039d380a6bacdb8e43" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 44409 } }
I20260928 16:39:40.747004 847 raft_consensus.cc:348] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53637373b524375834e5110de5e7587" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 43829 } } peers { permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 } } peers { permanent_uuid: "83b2286be80340039d380a6bacdb8e43" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 44409 } }
I20260928 16:39:40.747267 847 raft_consensus.cc:374] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:40.747349 847 raft_consensus.cc:729] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b2286be80340039d380a6bacdb8e43, State: Initialized, Role: FOLLOWER
I20260928 16:39:40.747926 539 heartbeater.cc:503] Master 127.25.71.126:33855 was elected leader, sending a full tablet report...
I20260928 16:39:40.748529 851 tablet_bootstrap.cc:492] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db: No bootstrap required, opened a new log
I20260928 16:39:40.748775 851 ts_tablet_manager.cc:1397] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260928 16:39:40.749667 848 ts_tablet_manager.cc:1428] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260928 16:39:40.752202 847 consensus_queue.cc:260] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53637373b524375834e5110de5e7587" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 43829 } } peers { permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 } } peers { permanent_uuid: "83b2286be80340039d380a6bacdb8e43" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 44409 } }
I20260928 16:39:40.753971 847 ts_tablet_manager.cc:1428] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260928 16:39:40.754436 816 heartbeater.cc:503] Master 127.25.71.126:33855 was elected leader, sending a full tablet report...
I20260928 16:39:40.754459 851 raft_consensus.cc:348] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53637373b524375834e5110de5e7587" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 43829 } } peers { permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 } } peers { permanent_uuid: "83b2286be80340039d380a6bacdb8e43" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 44409 } }
I20260928 16:39:40.754933 851 raft_consensus.cc:374] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:40.754994 851 raft_consensus.cc:729] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d7f25c092a41cc93edb7430142e0db, State: Initialized, Role: FOLLOWER
I20260928 16:39:40.755650 851 consensus_queue.cc:260] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53637373b524375834e5110de5e7587" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 43829 } } peers { permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 } } peers { permanent_uuid: "83b2286be80340039d380a6bacdb8e43" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 44409 } }
I20260928 16:39:40.757323 678 heartbeater.cc:503] Master 127.25.71.126:33855 was elected leader, sending a full tablet report...
I20260928 16:39:40.757548 851 ts_tablet_manager.cc:1428] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db: Time spent starting tablet: real 0.009s user 0.007s sys 0.003s
I20260928 16:39:40.908924 853 raft_consensus.cc:482] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:39:40.909230 853 raft_consensus.cc:504] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53637373b524375834e5110de5e7587" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 43829 } } peers { permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 } } peers { permanent_uuid: "83b2286be80340039d380a6bacdb8e43" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 44409 } }
I20260928 16:39:40.910813 853 leader_election.cc:288] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0d7f25c092a41cc93edb7430142e0db (127.25.71.66:36905), 83b2286be80340039d380a6bacdb8e43 (127.25.71.67:44409)
W20260928 16:39:40.912680 818 tablet.cc:2369] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:39:40.915810 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.911417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:53431 (local address 127.25.71.67:44409)
0928 16:39:40.911554 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:40.911558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:40.911858 (+ 300us) server_negotiation.cc:407] Connection header received
0928 16:39:40.912034 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:40.912038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:40.912099 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:40.912168 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:40.912507 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.913404 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.914784 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.915209 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.915334 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.915406 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:40.915561 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 16:39:40.915657 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:39:40.915838 856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.911196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:44409 (local address 127.25.71.65:53431)
0928 16:39:40.911730 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:40.911745 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:40.911879 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:40.912192 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:40.912195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:40.912212 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:40.912395 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.912402 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.913570 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.913574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.914659 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.914670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.915334 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.915337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.915442 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.915467 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 16:39:40.915533 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 16:39:40.915635 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":8,"thread_start_us":312,"threads_started":1}
I20260928 16:39:40.915941 845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.911339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:35269 (local address 127.25.71.66:36905)
0928 16:39:40.911512 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:40.911517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:40.911937 (+ 420us) server_negotiation.cc:407] Connection header received
0928 16:39:40.912094 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:40.912098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:40.912143 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:40.912205 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:40.912560 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.913404 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.914901 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.915372 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.915492 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.915550 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:40.915715 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 16:39:40.915803 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:39:40.916397 857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.911222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:36905 (local address 127.25.71.65:35269)
0928 16:39:40.911829 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:40.911845 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:40.911986 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:40.912232 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:40.912234 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:40.912252 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:40.912440 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.912447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.913541 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.913544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.914746 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:40.914757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.915493 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.915497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:40.915597 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.915624 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:39:40.915693 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 16:39:40.915792 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":67,"thread_start_us":307,"threads_started":1}
I20260928 16:39:40.916606 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7207dec97cc04f72ae8cd118cc48d580" candidate_uuid: "a53637373b524375834e5110de5e7587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b2286be80340039d380a6bacdb8e43" is_pre_election: true
I20260928 16:39:40.917006 768 raft_consensus.cc:2454] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a53637373b524375834e5110de5e7587 in term 0.
I20260928 16:39:40.917057 630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7207dec97cc04f72ae8cd118cc48d580" candidate_uuid: "a53637373b524375834e5110de5e7587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d7f25c092a41cc93edb7430142e0db" is_pre_election: true
I20260928 16:39:40.917388 630 raft_consensus.cc:2454] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a53637373b524375834e5110de5e7587 in term 0.
I20260928 16:39:40.917562 425 leader_election.cc:302] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b2286be80340039d380a6bacdb8e43, a53637373b524375834e5110de5e7587; no voters:
I20260928 16:39:40.917891 853 raft_consensus.cc:2781] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:39:40.917986 853 raft_consensus.cc:482] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:39:40.918038 853 raft_consensus.cc:3037] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:40.919442 853 raft_consensus.cc:504] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53637373b524375834e5110de5e7587" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 43829 } } peers { permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 } } peers { permanent_uuid: "83b2286be80340039d380a6bacdb8e43" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 44409 } }
I20260928 16:39:40.919852 853 leader_election.cc:288] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 [CANDIDATE]: Term 1 election: Requested vote from peers b0d7f25c092a41cc93edb7430142e0db (127.25.71.66:36905), 83b2286be80340039d380a6bacdb8e43 (127.25.71.67:44409)
I20260928 16:39:40.920307 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7207dec97cc04f72ae8cd118cc48d580" candidate_uuid: "a53637373b524375834e5110de5e7587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b2286be80340039d380a6bacdb8e43"
I20260928 16:39:40.920346 630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7207dec97cc04f72ae8cd118cc48d580" candidate_uuid: "a53637373b524375834e5110de5e7587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d7f25c092a41cc93edb7430142e0db"
I20260928 16:39:40.920475 768 raft_consensus.cc:3037] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:40.920508 630 raft_consensus.cc:3037] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:40.921854 768 raft_consensus.cc:2454] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a53637373b524375834e5110de5e7587 in term 1.
I20260928 16:39:40.921864 630 raft_consensus.cc:2454] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a53637373b524375834e5110de5e7587 in term 1.
W20260928 16:39:40.922158 680 tablet.cc:2369] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:39:40.922272 423 leader_election.cc:302] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a53637373b524375834e5110de5e7587, b0d7f25c092a41cc93edb7430142e0db; no voters:
I20260928 16:39:40.922550 853 raft_consensus.cc:2781] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:40.922923 853 raft_consensus.cc:686] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 [term 1 LEADER]: Becoming Leader. State: Replica: a53637373b524375834e5110de5e7587, State: Running, Role: LEADER
I20260928 16:39:40.923418 853 consensus_queue.cc:237] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53637373b524375834e5110de5e7587" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 43829 } } peers { permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 } } peers { permanent_uuid: "83b2286be80340039d380a6bacdb8e43" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 44409 } }
W20260928 16:39:40.927042 541 tablet.cc:2369] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:39:40.927654 32027 catalog_manager.cc:5495] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 reported cstate change: term changed from 0 to 1, leader changed from <none> to a53637373b524375834e5110de5e7587 (127.25.71.65). New cstate: current_term: 1 leader_uuid: "a53637373b524375834e5110de5e7587" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53637373b524375834e5110de5e7587" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 43829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b2286be80340039d380a6bacdb8e43" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 44409 } health_report { overall_health: UNKNOWN } } }
I20260928 16:39:40.964908 842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:40.961325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57596 (local address 127.25.71.65:43829)
0928 16:39:40.961517 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:40.961522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:40.961536 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:39:40.961625 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:40.961628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:40.961684 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:40.961783 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:40.962030 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.962562 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.963427 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:40.963644 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:40.963728 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:40.964043 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:40.964513 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:40.964617 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:40.964695 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 16:39:40.964756 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 16:39:40.973418 630 raft_consensus.cc:1261] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db [term 1 FOLLOWER]: Refusing update from remote peer a53637373b524375834e5110de5e7587: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:40.973546 768 raft_consensus.cc:1261] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43 [term 1 FOLLOWER]: Refusing update from remote peer a53637373b524375834e5110de5e7587: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:40.974606 858 consensus_queue.cc:1035] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:40.974994 853 consensus_queue.cc:1035] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b2286be80340039d380a6bacdb8e43" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 44409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:40.987438 862 mvcc.cc:205] Tried to move back new op lower bound from 7334353227656503296 to 7334353227465560064. Current Snapshot: MvccSnapshot[applied={T|T < 7334353227656503296}]
I20260928 16:39:41.006148 32027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:41.006397 32027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:41.012166 32027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f350c795eedf4b9a91b21283d5e0f6d9: Sending DeleteTablet for 3 replicas of tablet 7207dec97cc04f72ae8cd118cc48d580
I20260928 16:39:41.013314 470 tablet_service.cc:1514] Processing DeleteTablet for tablet 7207dec97cc04f72ae8cd118cc48d580 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:41 UTC) from {username='oryx', principal='oryx/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:57590
I20260928 16:39:41.013531 610 tablet_service.cc:1514] Processing DeleteTablet for tablet 7207dec97cc04f72ae8cd118cc48d580 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:41 UTC) from {username='oryx', principal='oryx/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:39826
I20260928 16:39:41.014816 871 tablet_replica.cc:330] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587: stopping tablet replica
I20260928 16:39:41.014917 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 7207dec97cc04f72ae8cd118cc48d580 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:41 UTC) from {username='oryx', principal='oryx/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:50806
I20260928 16:39:41.015422 871 raft_consensus.cc:2229] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:39:41.015673 873 tablet_replica.cc:330] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43: stopping tablet replica
I20260928 16:39:41.016057 871 raft_consensus.cc:2258] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:41.016227 873 raft_consensus.cc:2229] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:41.016649 872 tablet_replica.cc:330] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db: stopping tablet replica
I20260928 16:39:41.016674 873 raft_consensus.cc:2258] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:41.017082 872 raft_consensus.cc:2229] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:41.017513 872 raft_consensus.cc:2258] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:41.019172 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 31996
I20260928 16:39:41.022163 871 ts_tablet_manager.cc:1905] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:41.022163 873 ts_tablet_manager.cc:1905] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:41.026183 871 ts_tablet_manager.cc:1918] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:39:41.026223 873 ts_tablet_manager.cc:1918] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:39:41.026363 873 log.cc:1196] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/7207dec97cc04f72ae8cd118cc48d580
I20260928 16:39:41.026363 871 log.cc:1196] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-0/wal/wals/7207dec97cc04f72ae8cd118cc48d580
I20260928 16:39:41.026787 873 ts_tablet_manager.cc:1939] T 7207dec97cc04f72ae8cd118cc48d580 P 83b2286be80340039d380a6bacdb8e43: Deleting consensus metadata
I20260928 16:39:41.028038 872 ts_tablet_manager.cc:1905] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:41.028530 871 ts_tablet_manager.cc:1939] T 7207dec97cc04f72ae8cd118cc48d580 P a53637373b524375834e5110de5e7587: Deleting consensus metadata
W20260928 16:39:41.028610 703 connection.cc:441] server connection from 127.0.0.1:50806 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50806 (request call id 1) could send its response
W20260928 16:39:41.030030 424 connection.cc:441] server connection from 127.0.0.1:57590 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57590 (request call id 1) could send its response
I20260928 16:39:41.034289 872 ts_tablet_manager.cc:1918] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:39:41.034439 872 log.cc:1196] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/7207dec97cc04f72ae8cd118cc48d580
I20260928 16:39:41.034855 872 ts_tablet_manager.cc:1939] T 7207dec97cc04f72ae8cd118cc48d580 P b0d7f25c092a41cc93edb7430142e0db: Deleting consensus metadata
W20260928 16:39:41.038301 562 connection.cc:441] server connection from 127.0.0.1:39826 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39826 (request call id 1) could send its response
I20260928 16:39:41.042788 829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:41.018565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46810 (local address 127.25.71.125:44555)
0928 16:39:41.018869 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:41.018874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:41.018893 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:39:41.018955 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:41.018958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:41.019040 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:41.019169 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:41.031209 (+ 12040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:41.032280 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:41.033635 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:41.034077 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:41.034224 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:41.036908 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:41.036913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:41.036916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:41.036952 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:41.038775 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:41.040129 (+ 1354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:41.040133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:41.040135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:41.040526 (+ 391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:41.040863 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:41.040866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:41.040868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:41.041131 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:41.041264 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:41.042328 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0928 16:39:41.042491 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260928 16:39:41.045892 32755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:41.034716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.25.71.125:44555)
0928 16:39:41.034855 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:41.034860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:41.035886 (+ 1026us) server_negotiation.cc:407] Connection header received
0928 16:39:41.036095 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:41.036110 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:41.036168 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:41.036214 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:41.037180 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:41.037960 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:41.039269 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:41.039594 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:41.039693 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:41.041508 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:41.041513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:41.041516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:41.041549 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:41.043429 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:41.043951 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:41.043955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:41.043957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:41.044045 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:41.044368 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:41.044371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:41.044373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:41.044982 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:41.045125 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:41.045549 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 16:39:41.045694 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:39:41.050036 684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:41.035680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55414 (local address 127.25.71.124:38021)
0928 16:39:41.035821 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:41.035827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:41.038902 (+ 3075us) server_negotiation.cc:407] Connection header received
0928 16:39:41.039115 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:41.039119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:41.039188 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:41.039248 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:41.040299 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:41.041147 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:41.042304 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:41.042815 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:41.042911 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:41.045856 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:41.045861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:41.045864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:41.045899 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:41.047979 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:41.048397 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:41.048401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:41.048403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:41.048506 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:41.048870 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:41.048874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:41.048876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:41.049128 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:41.049249 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:41.049638 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 16:39:41.049813 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:39:41.053078 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:41.018405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55400 (local address 127.25.71.124:38021)
0928 16:39:41.018755 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:41.018761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:41.043511 (+ 24750us) server_negotiation.cc:407] Connection header received
0928 16:39:41.043704 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:41.043707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:41.043769 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:41.043824 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:41.045101 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:41.045942 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:41.047526 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:41.047913 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:41.047993 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:41.049493 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:41.049497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:41.049500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:41.049524 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:41.051060 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:41.051549 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:41.051553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:41.051555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:41.051712 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:41.052018 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:41.052021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:41.052023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:41.052255 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:41.052421 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:41.052728 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 16:39:41.052890 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247}
I20260928 16:39:41.990059 881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:41.989558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.66:36739)
0928 16:39:41.989850 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:41.989953 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":84,"threads_started":1}
W20260928 16:39:41.990526 678 heartbeater.cc:644] Failed to heartbeat to 127.25.71.126:33855 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
I20260928 16:39:41.992980 882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:41.992338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.65:34749)
0928 16:39:41.992742 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:41.992858 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":98,"threads_started":1}
W20260928 16:39:41.993535 539 heartbeater.cc:644] Failed to heartbeat to 127.25.71.126:33855 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
I20260928 16:39:41.995584 883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:41.995016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.67:36867)
0928 16:39:41.995329 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:41.995445 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":92,"threads_started":1}
W20260928 16:39:41.996024 816 heartbeater.cc:644] Failed to heartbeat to 127.25.71.126:33855 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
I20260928 16:39:42.513389 884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f350c795eedf4b9a91b21283d5e0f6d9)
I20260928 16:39:42.513633 884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: false } }
I20260928 16:39:42.514384 885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f350c795eedf4b9a91b21283d5e0f6d9)
I20260928 16:39:42.514700 885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: false } }
I20260928 16:39:42.514876 884 leader_election.cc:288] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f350c795eedf4b9a91b21283d5e0f6d9 (127.25.71.126:33855), 91facbd2cee44b1f814eb251c69daf0a (127.25.71.125:44555)
I20260928 16:39:42.515851 887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.515170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.0.0.1:56798)
0928 16:39:42.515594 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.515724 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":111,"threads_started":1}
I20260928 16:39:42.515977 885 leader_election.cc:288] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f350c795eedf4b9a91b21283d5e0f6d9 (127.25.71.126:33855), 2b1cd9df519e432b883536905633b131 (127.25.71.124:38021)
W20260928 16:39:42.516089 341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
I20260928 16:39:42.516646 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.516080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.0.0.1:56806)
0928 16:39:42.516469 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.516548 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":69,"threads_started":1}
W20260928 16:39:42.516954 32618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
I20260928 16:39:42.517145 887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.516887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.0.0.1:56808)
0928 16:39:42.517034 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.517081 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260928 16:39:42.517262 341 leader_election.cc:334] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f350c795eedf4b9a91b21283d5e0f6d9 (127.25.71.126:33855): Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
I20260928 16:39:42.518070 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.517783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.0.0.1:56816)
0928 16:39:42.517909 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.517973 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 16:39:42.518299 32618 leader_election.cc:334] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f350c795eedf4b9a91b21283d5e0f6d9 (127.25.71.126:33855): Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
I20260928 16:39:42.519979 888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.515332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46820 (local address 127.25.71.125:44555)
0928 16:39:42.515617 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:42.515622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:42.515709 (+ 87us) server_negotiation.cc:407] Connection header received
0928 16:39:42.515873 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:42.515875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:42.515922 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:42.516000 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:42.516382 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.517140 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.518604 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.519366 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.519460 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:42.519520 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:42.519763 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 16:39:42.519852 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":207,"thread_start_us":58,"threads_started":1}
I20260928 16:39:42.520112 886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.515196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:44555 (local address 127.0.0.1:46820)
0928 16:39:42.515615 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.515631 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:42.515762 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:42.516128 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:42.516131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:42.516151 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:42.516255 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:42.516261 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.517256 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.517259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:42.518468 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:42.518479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.519508 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.519512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:42.519612 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:42.519644 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 16:39:42.519810 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 16:39:42.519921 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":4,"thread_start_us":95,"threads_started":1}
I20260928 16:39:42.520694 32643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b1cd9df519e432b883536905633b131" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "91facbd2cee44b1f814eb251c69daf0a" is_pre_election: true
I20260928 16:39:42.520982 32643 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1cd9df519e432b883536905633b131 in term 1.
I20260928 16:39:42.521093 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.516763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55418 (local address 127.25.71.124:38021)
0928 16:39:42.517274 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:42.517282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:42.517293 (+ 11us) server_negotiation.cc:407] Connection header received
0928 16:39:42.517323 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:42.517325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:42.517375 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:42.517443 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:42.517777 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.518525 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.519832 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.520518 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.520625 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:42.520697 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:42.520871 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 16:39:42.520959 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":445,"thread_start_us":94,"threads_started":1}
I20260928 16:39:42.521142 891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.516409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:38021 (local address 127.0.0.1:55418)
0928 16:39:42.517010 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.517024 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:42.517100 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:42.517536 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:42.517539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:42.517555 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:42.517632 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:42.517637 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.518697 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.518700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:42.519713 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:42.519724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.520648 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.520651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:42.520723 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:42.520746 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:39:42.520813 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 16:39:42.520917 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":73,"threads_started":1}
I20260928 16:39:42.521574 343 leader_election.cc:302] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b1cd9df519e432b883536905633b131, 91facbd2cee44b1f814eb251c69daf0a; no voters: f350c795eedf4b9a91b21283d5e0f6d9
I20260928 16:39:42.521874 884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 16:39:42.521840 369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91facbd2cee44b1f814eb251c69daf0a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2b1cd9df519e432b883536905633b131" is_pre_election: true
I20260928 16:39:42.521938 884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f350c795eedf4b9a91b21283d5e0f6d9)
I20260928 16:39:42.521975 884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:39:42.523449 884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: false } }
I20260928 16:39:42.523743 369 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 91facbd2cee44b1f814eb251c69daf0a in current term 2: Already voted for candidate 2b1cd9df519e432b883536905633b131 in this term.
I20260928 16:39:42.523947 884 leader_election.cc:288] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [CANDIDATE]: Term 2 election: Requested vote from peers f350c795eedf4b9a91b21283d5e0f6d9 (127.25.71.126:33855), 91facbd2cee44b1f814eb251c69daf0a (127.25.71.125:44555)
I20260928 16:39:42.524459 886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.524127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.0.0.1:56830)
0928 16:39:42.524271 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.524377 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 16:39:42.524380 32643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b1cd9df519e432b883536905633b131" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "91facbd2cee44b1f814eb251c69daf0a"
I20260928 16:39:42.524565 32643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:39:42.524629 32618 leader_election.cc:302] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91facbd2cee44b1f814eb251c69daf0a; no voters: 2b1cd9df519e432b883536905633b131, f350c795eedf4b9a91b21283d5e0f6d9
I20260928 16:39:42.525333 886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.525113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.0.0.1:56846)
0928 16:39:42.525226 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.525279 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 16:39:42.525643 341 leader_election.cc:334] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f350c795eedf4b9a91b21283d5e0f6d9 (127.25.71.126:33855): Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
I20260928 16:39:42.526383 32643 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b1cd9df519e432b883536905633b131 in term 2.
I20260928 16:39:42.526541 885 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:39:42.526813 343 leader_election.cc:302] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b1cd9df519e432b883536905633b131, 91facbd2cee44b1f814eb251c69daf0a; no voters: f350c795eedf4b9a91b21283d5e0f6d9
I20260928 16:39:42.527070 884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:39:42.527386 884 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [term 2 LEADER]: Becoming Leader. State: Replica: 2b1cd9df519e432b883536905633b131, State: Running, Role: LEADER
I20260928 16:39:42.527722 884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: false } }
I20260928 16:39:42.529255 894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b1cd9df519e432b883536905633b131. Latest consensus state: current_term: 2 leader_uuid: "2b1cd9df519e432b883536905633b131" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: false } } }
I20260928 16:39:42.529474 894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:42.529826 896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:39:42.532349 896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:39:42.533179 896 catalog_manager.cc:1259] Loaded cluster ID: 521ae1f785a849e1a4661590fe3180c4
I20260928 16:39:42.533245 896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:39:42.533862 896 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:39:42.534297 896 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: Loaded TSK: 0
I20260928 16:39:42.535108 896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:39:42.593194 359 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:39:42.596668 359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:39:42.600512 886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.600208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.0.0.1:56872)
0928 16:39:42.600343 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.600409 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 16:39:42.600610 32643 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [term 2 FOLLOWER]: Refusing update from remote peer 2b1cd9df519e432b883536905633b131: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 16:39:42.601406 886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.601150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.0.0.1:56882)
0928 16:39:42.601274 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.601338 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 16:39:42.601275 894 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 16:39:42.601594 341 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 -> Peer f350c795eedf4b9a91b21283d5e0f6d9 (127.25.71.126:33855): Couldn't send request to peer f350c795eedf4b9a91b21283d5e0f6d9. Status: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 16:39:42.603271 885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b1cd9df519e432b883536905633b131. Latest consensus state: current_term: 2 leader_uuid: "2b1cd9df519e432b883536905633b131" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: false } } }
I20260928 16:39:42.603461 885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:39:42.605345 885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2b1cd9df519e432b883536905633b131" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: false } } }
I20260928 16:39:42.605474 885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91facbd2cee44b1f814eb251c69daf0a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:39:42.605156 884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "2b1cd9df519e432b883536905633b131" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f350c795eedf4b9a91b21283d5e0f6d9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 33855 } } peers { permanent_uuid: "91facbd2cee44b1f814eb251c69daf0a" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 44555 } attrs { promote: false } } peers { permanent_uuid: "2b1cd9df519e432b883536905633b131" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38021 } attrs { promote: false } } }
I20260928 16:39:42.605772 884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:42.617856 886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.613366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:44409 (local address 127.0.0.1:50808)
0928 16:39:42.613489 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.613506 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:42.613583 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:42.614269 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:42.614273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:42.614296 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:42.614436 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:42.614447 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.615652 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.615655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:42.616660 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:42.616671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.617385 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.617388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:42.617474 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:42.617499 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 16:39:42.617568 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 16:39:42.617679 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":4}
I20260928 16:39:42.617874 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.613566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50808 (local address 127.25.71.67:44409)
0928 16:39:42.613918 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:42.613923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:42.613948 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:39:42.614003 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:42.614006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:42.614058 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:42.614181 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:42.614622 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.615516 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.616813 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.617257 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.617352 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:42.617439 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:42.617595 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 16:39:42.617681 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":132,"threads_started":1}
I20260928 16:39:42.618736 887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.613610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:43829 (local address 127.0.0.1:57604)
0928 16:39:42.613848 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.613864 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:42.613938 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:42.614929 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:42.614932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:42.614949 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:42.615081 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:42.615089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.616028 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.616032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:42.617387 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:42.617399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.618293 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.618297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:42.618408 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:42.618432 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:39:42.618493 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 16:39:42.618573 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":92}
I20260928 16:39:42.618726 748 tablet_service.cc:1467] Processing CreateTablet for tablet 00329ad192f34e0cbf27c4174f190772 (DEFAULT_TABLE table=test-table [id=b10b012913d740019726b1db9b4d4de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:42.619122 748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00329ad192f34e0cbf27c4174f190772. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:42.619166 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.614151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:36905 (local address 127.0.0.1:39836)
0928 16:39:42.614417 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.614429 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:42.614511 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:42.614998 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:42.615001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:42.615020 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:42.615117 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:42.615123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.616151 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.616155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:42.617393 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:42.617399 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.618727 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.618730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:42.618827 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:42.618854 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:39:42.618906 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 16:39:42.618996 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":3,"thread_start_us":138,"threads_started":1}
I20260928 16:39:42.619297 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.614486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57604 (local address 127.25.71.65:43829)
0928 16:39:42.614722 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:42.614725 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:42.614736 (+ 11us) server_negotiation.cc:407] Connection header received
0928 16:39:42.614779 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:42.614781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:42.614826 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:42.614899 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:42.615211 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.615926 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.617545 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.618173 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.618279 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:42.619064 (+ 785us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:42.619104 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 16:39:42.619180 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":83,"threads_started":1}
I20260928 16:39:42.620141 470 tablet_service.cc:1467] Processing CreateTablet for tablet 00329ad192f34e0cbf27c4174f190772 (DEFAULT_TABLE table=test-table [id=b10b012913d740019726b1db9b4d4de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:42.620352 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.614333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39836 (local address 127.25.71.66:36905)
0928 16:39:42.614723 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:42.614729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:42.614743 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:39:42.614800 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:42.614803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:42.614856 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:42.615224 (+ 368us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:42.615329 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.616039 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.617566 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.618567 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.619997 (+ 1430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:42.620098 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:42.620134 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 16:39:42.620212 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":114,"threads_started":1}
I20260928 16:39:42.620471 470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00329ad192f34e0cbf27c4174f190772. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:42.622949 906 tablet_bootstrap.cc:492] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43: Bootstrap starting.
I20260928 16:39:42.623839 907 tablet_bootstrap.cc:492] T 00329ad192f34e0cbf27c4174f190772 P a53637373b524375834e5110de5e7587: Bootstrap starting.
I20260928 16:39:42.624243 906 tablet_bootstrap.cc:654] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:42.624536 610 tablet_service.cc:1467] Processing CreateTablet for tablet 00329ad192f34e0cbf27c4174f190772 (DEFAULT_TABLE table=test-table [id=b10b012913d740019726b1db9b4d4de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:42.624861 610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00329ad192f34e0cbf27c4174f190772. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:42.624953 907 tablet_bootstrap.cc:654] T 00329ad192f34e0cbf27c4174f190772 P a53637373b524375834e5110de5e7587: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:42.625730 906 tablet_bootstrap.cc:492] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43: No bootstrap required, opened a new log
I20260928 16:39:42.625818 906 ts_tablet_manager.cc:1397] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:39:42.626186 907 tablet_bootstrap.cc:492] T 00329ad192f34e0cbf27c4174f190772 P a53637373b524375834e5110de5e7587: No bootstrap required, opened a new log
I20260928 16:39:42.626263 907 ts_tablet_manager.cc:1397] T 00329ad192f34e0cbf27c4174f190772 P a53637373b524375834e5110de5e7587: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:39:42.626317 906 raft_consensus.cc:348] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b2286be80340039d380a6bacdb8e43" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 44409 } } peers { permanent_uuid: "a53637373b524375834e5110de5e7587" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 43829 } } peers { permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 } }
I20260928 16:39:42.626448 906 raft_consensus.cc:374] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:42.626480 906 raft_consensus.cc:729] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b2286be80340039d380a6bacdb8e43, State: Initialized, Role: FOLLOWER
I20260928 16:39:42.626662 906 consensus_queue.cc:260] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b2286be80340039d380a6bacdb8e43" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 44409 } } peers { permanent_uuid: "a53637373b524375834e5110de5e7587" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 43829 } } peers { permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 } }
I20260928 16:39:42.626693 907 raft_consensus.cc:348] T 00329ad192f34e0cbf27c4174f190772 P a53637373b524375834e5110de5e7587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b2286be80340039d380a6bacdb8e43" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 44409 } } peers { permanent_uuid: "a53637373b524375834e5110de5e7587" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 43829 } } peers { permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 } }
I20260928 16:39:42.626809 907 raft_consensus.cc:374] T 00329ad192f34e0cbf27c4174f190772 P a53637373b524375834e5110de5e7587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:42.626828 907 raft_consensus.cc:729] T 00329ad192f34e0cbf27c4174f190772 P a53637373b524375834e5110de5e7587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a53637373b524375834e5110de5e7587, State: Initialized, Role: FOLLOWER
I20260928 16:39:42.627002 907 consensus_queue.cc:260] T 00329ad192f34e0cbf27c4174f190772 P a53637373b524375834e5110de5e7587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b2286be80340039d380a6bacdb8e43" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 44409 } } peers { permanent_uuid: "a53637373b524375834e5110de5e7587" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 43829 } } peers { permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 } }
I20260928 16:39:42.627061 906 ts_tablet_manager.cc:1428] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 16:39:42.627374 907 ts_tablet_manager.cc:1428] T 00329ad192f34e0cbf27c4174f190772 P a53637373b524375834e5110de5e7587: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:39:42.628619 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.628151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.65:41439)
0928 16:39:42.628414 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.628489 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":103,"threads_started":1}
I20260928 16:39:42.629797 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.629505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.65:53121)
0928 16:39:42.629674 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.629724 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 16:39:42.632056 912 tablet_bootstrap.cc:492] T 00329ad192f34e0cbf27c4174f190772 P b0d7f25c092a41cc93edb7430142e0db: Bootstrap starting.
I20260928 16:39:42.633008 912 tablet_bootstrap.cc:654] T 00329ad192f34e0cbf27c4174f190772 P b0d7f25c092a41cc93edb7430142e0db: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:42.634236 912 tablet_bootstrap.cc:492] T 00329ad192f34e0cbf27c4174f190772 P b0d7f25c092a41cc93edb7430142e0db: No bootstrap required, opened a new log
I20260928 16:39:42.634330 912 ts_tablet_manager.cc:1397] T 00329ad192f34e0cbf27c4174f190772 P b0d7f25c092a41cc93edb7430142e0db: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:39:42.634796 912 raft_consensus.cc:348] T 00329ad192f34e0cbf27c4174f190772 P b0d7f25c092a41cc93edb7430142e0db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b2286be80340039d380a6bacdb8e43" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 44409 } } peers { permanent_uuid: "a53637373b524375834e5110de5e7587" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 43829 } } peers { permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 } }
I20260928 16:39:42.634917 912 raft_consensus.cc:374] T 00329ad192f34e0cbf27c4174f190772 P b0d7f25c092a41cc93edb7430142e0db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:42.634948 912 raft_consensus.cc:729] T 00329ad192f34e0cbf27c4174f190772 P b0d7f25c092a41cc93edb7430142e0db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d7f25c092a41cc93edb7430142e0db, State: Initialized, Role: FOLLOWER
I20260928 16:39:42.634898 538 heartbeater.cc:503] Master 127.25.71.124:38021 was elected leader, sending a full tablet report...
I20260928 16:39:42.635138 912 consensus_queue.cc:260] T 00329ad192f34e0cbf27c4174f190772 P b0d7f25c092a41cc93edb7430142e0db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b2286be80340039d380a6bacdb8e43" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 44409 } } peers { permanent_uuid: "a53637373b524375834e5110de5e7587" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 43829 } } peers { permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 } }
I20260928 16:39:42.636443 912 ts_tablet_manager.cc:1428] T 00329ad192f34e0cbf27c4174f190772 P b0d7f25c092a41cc93edb7430142e0db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:39:42.637197 917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.636232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.66:37221)
0928 16:39:42.636968 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.637055 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":154,"threads_started":1}
I20260928 16:39:42.638293 677 heartbeater.cc:503] Master 127.25.71.124:38021 was elected leader, sending a full tablet report...
I20260928 16:39:42.638319 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.627577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.67:44149)
0928 16:39:42.633681 (+ 6104us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.633758 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6017,"thread_start_us":97,"threads_started":1}
I20260928 16:39:42.639214 917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.637847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.66:41609)
0928 16:39:42.639068 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.639127 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1144,"thread_start_us":126,"threads_started":1}
I20260928 16:39:42.639880 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.639236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.67:60259)
0928 16:39:42.639728 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.639796 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405}
I20260928 16:39:42.676443 910 raft_consensus.cc:482] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:39:42.676607 910 raft_consensus.cc:504] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b2286be80340039d380a6bacdb8e43" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 44409 } } peers { permanent_uuid: "a53637373b524375834e5110de5e7587" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 43829 } } peers { permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 } }
I20260928 16:39:42.677928 910 leader_election.cc:288] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a53637373b524375834e5110de5e7587 (127.25.71.65:43829), b0d7f25c092a41cc93edb7430142e0db (127.25.71.66:36905)
I20260928 16:39:42.684303 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.678363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:53029 (local address 127.25.71.66:36905)
0928 16:39:42.678491 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:42.678497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:42.678556 (+ 59us) server_negotiation.cc:407] Connection header received
0928 16:39:42.678850 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:42.678853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:42.678923 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:42.679021 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:42.679744 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.680671 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.682655 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.683403 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.683566 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:42.683638 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:42.684024 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 16:39:42.684127 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 16:39:42.684310 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.678789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:57631 (local address 127.25.71.65:43829)
0928 16:39:42.678970 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:42.678975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:42.678989 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:39:42.679049 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:42.679052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:42.679102 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:42.679176 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:42.679745 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.680619 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.682655 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.683410 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.683566 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:42.683643 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:42.684024 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 16:39:42.684127 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 16:39:42.684320 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.678239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:36905 (local address 127.25.71.67:53029)
0928 16:39:42.678459 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.678476 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:42.678670 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:42.679304 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:42.679307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:42.679338 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:42.679524 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:42.679532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.680820 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.680824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:42.682436 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:42.682452 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.683566 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.683570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:42.683754 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:42.683896 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 16:39:42.683992 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 16:39:42.684096 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":145,"spinlock_wait_cycles":5248}
I20260928 16:39:42.684320 919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.678297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:43829 (local address 127.25.71.67:57631)
0928 16:39:42.678829 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.678843 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:42.678930 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:42.679304 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:42.679307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:42.679331 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:42.679527 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:42.679532 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.680859 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.680862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:42.682438 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:42.682451 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.683566 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.683570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:42.683825 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:42.683900 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 16:39:42.683992 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 16:39:42.684096 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":398,"spinlock_wait_cycles":2048,"thread_start_us":206,"threads_started":1}
I20260928 16:39:42.685199 630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00329ad192f34e0cbf27c4174f190772" candidate_uuid: "83b2286be80340039d380a6bacdb8e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d7f25c092a41cc93edb7430142e0db" is_pre_election: true
I20260928 16:39:42.685343 490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00329ad192f34e0cbf27c4174f190772" candidate_uuid: "83b2286be80340039d380a6bacdb8e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a53637373b524375834e5110de5e7587" is_pre_election: true
I20260928 16:39:42.685425 630 raft_consensus.cc:2454] T 00329ad192f34e0cbf27c4174f190772 P b0d7f25c092a41cc93edb7430142e0db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b2286be80340039d380a6bacdb8e43 in term 0.
I20260928 16:39:42.685631 490 raft_consensus.cc:2454] T 00329ad192f34e0cbf27c4174f190772 P a53637373b524375834e5110de5e7587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b2286be80340039d380a6bacdb8e43 in term 0.
I20260928 16:39:42.685905 701 leader_election.cc:302] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b2286be80340039d380a6bacdb8e43, b0d7f25c092a41cc93edb7430142e0db; no voters:
I20260928 16:39:42.686328 910 raft_consensus.cc:2781] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:39:42.686405 910 raft_consensus.cc:482] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:39:42.686434 910 raft_consensus.cc:3037] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:42.687865 910 raft_consensus.cc:504] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b2286be80340039d380a6bacdb8e43" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 44409 } } peers { permanent_uuid: "a53637373b524375834e5110de5e7587" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 43829 } } peers { permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 } }
I20260928 16:39:42.688365 910 leader_election.cc:288] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 [CANDIDATE]: Term 1 election: Requested vote from peers a53637373b524375834e5110de5e7587 (127.25.71.65:43829), b0d7f25c092a41cc93edb7430142e0db (127.25.71.66:36905)
I20260928 16:39:42.689085 630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00329ad192f34e0cbf27c4174f190772" candidate_uuid: "83b2286be80340039d380a6bacdb8e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d7f25c092a41cc93edb7430142e0db"
I20260928 16:39:42.689132 490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00329ad192f34e0cbf27c4174f190772" candidate_uuid: "83b2286be80340039d380a6bacdb8e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a53637373b524375834e5110de5e7587"
I20260928 16:39:42.689285 630 raft_consensus.cc:3037] T 00329ad192f34e0cbf27c4174f190772 P b0d7f25c092a41cc93edb7430142e0db [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:42.689339 490 raft_consensus.cc:3037] T 00329ad192f34e0cbf27c4174f190772 P a53637373b524375834e5110de5e7587 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:42.690441 630 raft_consensus.cc:2454] T 00329ad192f34e0cbf27c4174f190772 P b0d7f25c092a41cc93edb7430142e0db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b2286be80340039d380a6bacdb8e43 in term 1.
I20260928 16:39:42.690587 490 raft_consensus.cc:2454] T 00329ad192f34e0cbf27c4174f190772 P a53637373b524375834e5110de5e7587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b2286be80340039d380a6bacdb8e43 in term 1.
I20260928 16:39:42.690867 701 leader_election.cc:302] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b2286be80340039d380a6bacdb8e43, b0d7f25c092a41cc93edb7430142e0db; no voters:
I20260928 16:39:42.691092 910 raft_consensus.cc:2781] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:42.691376 910 raft_consensus.cc:686] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 [term 1 LEADER]: Becoming Leader. State: Replica: 83b2286be80340039d380a6bacdb8e43, State: Running, Role: LEADER
I20260928 16:39:42.691613 815 heartbeater.cc:503] Master 127.25.71.124:38021 was elected leader, sending a full tablet report...
I20260928 16:39:42.691707 910 consensus_queue.cc:237] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b2286be80340039d380a6bacdb8e43" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 44409 } } peers { permanent_uuid: "a53637373b524375834e5110de5e7587" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 43829 } } peers { permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 } }
I20260928 16:39:42.692440 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.692176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.67:59279)
0928 16:39:42.692305 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.692366 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 16:39:42.694855 359 catalog_manager.cc:5495] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b2286be80340039d380a6bacdb8e43 (127.25.71.67). New cstate: current_term: 1 leader_uuid: "83b2286be80340039d380a6bacdb8e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b2286be80340039d380a6bacdb8e43" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 44409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a53637373b524375834e5110de5e7587" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 43829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 } health_report { overall_health: UNKNOWN } } }
I20260928 16:39:42.726346 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.721749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.25.71.67:44409)
0928 16:39:42.721953 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:42.721958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:42.721976 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:39:42.722154 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:42.722157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:42.722233 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:42.722351 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:42.722717 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.723298 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.724447 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:42.724687 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:42.724786 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:42.725156 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:42.725776 (+ 620us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:42.725880 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:42.726049 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 16:39:42.726121 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 16:39:42.730099 630 raft_consensus.cc:1261] T 00329ad192f34e0cbf27c4174f190772 P b0d7f25c092a41cc93edb7430142e0db [term 1 FOLLOWER]: Refusing update from remote peer 83b2286be80340039d380a6bacdb8e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:42.730515 490 raft_consensus.cc:1261] T 00329ad192f34e0cbf27c4174f190772 P a53637373b524375834e5110de5e7587 [term 1 FOLLOWER]: Refusing update from remote peer 83b2286be80340039d380a6bacdb8e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:42.730762 920 consensus_queue.cc:1035] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0d7f25c092a41cc93edb7430142e0db" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:42.731293 920 consensus_queue.cc:1035] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a53637373b524375834e5110de5e7587" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 43829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:42.733021 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.732643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.66:43955)
0928 16:39:42.732815 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.732891 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 16:39:42.736073 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.735619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.65:33167)
0928 16:39:42.735866 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.735955 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260928 16:39:42.737850 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.737536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.65:54779)
0928 16:39:42.737689 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.737761 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 16:39:42.738355 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.737959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.67:51253)
0928 16:39:42.738199 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.738262 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260928 16:39:42.738906 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.738507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.66:34643)
0928 16:39:42.738770 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.738822 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260928 16:39:42.739725 925 mvcc.cc:205] Tried to move back new op lower bound from 7334353234856026112 to 7334353234708369408. Current Snapshot: MvccSnapshot[applied={T|T < 7334353234856026112}]
I20260928 16:39:42.742128 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.741741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:33855 (local address 127.25.71.67:45673)
0928 16:39:42.741981 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.742039 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260928 16:39:42.742586 923 mvcc.cc:205] Tried to move back new op lower bound from 7334353234856026112 to 7334353234708369408. Current Snapshot: MvccSnapshot[applied={T|T < 7334353234856026112 or (T in {7334353234856026112})}]
I20260928 16:39:42.748119 358 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:42.748370 358 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:42.754199 358 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b1cd9df519e432b883536905633b131: Sending DeleteTablet for 3 replicas of tablet 00329ad192f34e0cbf27c4174f190772
I20260928 16:39:42.755347 470 tablet_service.cc:1514] Processing DeleteTablet for tablet 00329ad192f34e0cbf27c4174f190772 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:42 UTC) from {username='oryx', principal='oryx/127.25.71.124@KRBTEST.COM'} at 127.0.0.1:57604
I20260928 16:39:42.755410 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 00329ad192f34e0cbf27c4174f190772 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:42 UTC) from {username='oryx', principal='oryx/127.25.71.124@KRBTEST.COM'} at 127.0.0.1:50808
I20260928 16:39:42.755849 933 tablet_replica.cc:330] T 00329ad192f34e0cbf27c4174f190772 P a53637373b524375834e5110de5e7587: stopping tablet replica
I20260928 16:39:42.755923 610 tablet_service.cc:1514] Processing DeleteTablet for tablet 00329ad192f34e0cbf27c4174f190772 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:42 UTC) from {username='oryx', principal='oryx/127.25.71.124@KRBTEST.COM'} at 127.0.0.1:39836
I20260928 16:39:42.756009 933 raft_consensus.cc:2229] T 00329ad192f34e0cbf27c4174f190772 P a53637373b524375834e5110de5e7587 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:42.755999 934 tablet_replica.cc:330] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43: stopping tablet replica
I20260928 16:39:42.756194 933 raft_consensus.cc:2258] T 00329ad192f34e0cbf27c4174f190772 P a53637373b524375834e5110de5e7587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:42.756390 934 raft_consensus.cc:2229] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:39:42.756948 934 raft_consensus.cc:2258] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:42.757134 933 ts_tablet_manager.cc:1905] T 00329ad192f34e0cbf27c4174f190772 P a53637373b524375834e5110de5e7587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:42.757566 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 309
I20260928 16:39:42.759677 934 ts_tablet_manager.cc:1905] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:42.762797 934 ts_tablet_manager.cc:1918] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:39:42.762904 934 log.cc:1196] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/00329ad192f34e0cbf27c4174f190772
I20260928 16:39:42.763185 934 ts_tablet_manager.cc:1939] T 00329ad192f34e0cbf27c4174f190772 P 83b2286be80340039d380a6bacdb8e43: Deleting consensus metadata
I20260928 16:39:42.763214 935 tablet_replica.cc:330] T 00329ad192f34e0cbf27c4174f190772 P b0d7f25c092a41cc93edb7430142e0db: stopping tablet replica
I20260928 16:39:42.763526 935 raft_consensus.cc:2229] T 00329ad192f34e0cbf27c4174f190772 P b0d7f25c092a41cc93edb7430142e0db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:42.763782 935 raft_consensus.cc:2258] T 00329ad192f34e0cbf27c4174f190772 P b0d7f25c092a41cc93edb7430142e0db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:42.764163 342 catalog_manager.cc:4841] TS 83b2286be80340039d380a6bacdb8e43 (127.25.71.67:44409): tablet 00329ad192f34e0cbf27c4174f190772 (table test-table [id=b10b012913d740019726b1db9b4d4de9]) successfully deleted
I20260928 16:39:42.765537 935 ts_tablet_manager.cc:1905] T 00329ad192f34e0cbf27c4174f190772 P b0d7f25c092a41cc93edb7430142e0db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:42.768296 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 546
I20260928 16:39:42.769703 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.769302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:43829 (local address 127.0.0.1:57620)
0928 16:39:42.769523 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.769582 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.65:43829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260928 16:39:42.770105 342 catalog_manager.cc:4584] TS a53637373b524375834e5110de5e7587 (127.25.71.65:43829): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 00329ad192f34e0cbf27c4174f190772: Network error: Client connection negotiation failed: client connection to 127.25.71.65:43829: connect: Connection refused (error 111)
I20260928 16:39:42.776870 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 685
I20260928 16:39:42.776948 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:42.776609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:36905 (local address 127.0.0.1:39838)
0928 16:39:42.776753 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:42.776825 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.66:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 16:39:42.784271 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 32496
I20260928 16:39:42.790359 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 32597
2026-09-28T16:39:42Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5877 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:39:42 dist-test-slave-nqgh krb5kdc[938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:39:42 dist-test-slave-nqgh krb5kdc[938](info): set up 2 sockets
Sep 28 16:39:42 dist-test-slave-nqgh krb5kdc[938](info): commencing operation
krb5kdc: starting...
W20260928 16:39:43.769556 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:39:43 dist-test-slave-nqgh krb5kdc[938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:39:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:39:43Z Disabled control of system clock
WARNING: no policy specified for oryx/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.126.keytab.
Entry for principal oryx/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.126.keytab.
I20260928 16:39:43.919165 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:36649
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.126.keytab
--principal=oryx/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:42927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:36649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:44.052021 952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:44.052424 952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:44.052495 952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:44.057271 952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:44.057380 952 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 16:39:44.057408 952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:44.057431 952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:44.057454 952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:44.057478 952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:39:44.063951 952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:42927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:36649
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.126.keytab
--principal=oryx/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:36649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:44.065507 952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:44.066876 952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:44.077872 958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:44.077857 957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:44.077857 960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:44.078097 952 server_base.cc:1034] running on GCE node
I20260928 16:39:44.079229 952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:44.080612 952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:44.081910 952 hybrid_clock.cc:648] HybridClock initialized: now 1790613584081874 us; error 72 us; skew 500 ppm
Sep 28 16:39:44 dist-test-slave-nqgh krb5kdc[938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613584, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:44.091466 952 init.cc:377] Logged in from keytab as oryx/127.25.71.126@KRBTEST.COM (short username oryx)
I20260928 16:39:44.092715 952 webserver.cc:460] Webserver started at http://127.25.71.126:33951/ using document root <none> and password file <none>
I20260928 16:39:44.093369 952 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:44.093448 952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:44.093664 952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:44.096061 952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "c146d4e873164b5e8d65d04f6d9a460d"
format_stamp: "Formatted at 2026-09-28 16:39:44 on dist-test-slave-nqgh"
server_key: "e1a740399df7f85152a51707cfc5d54b"
server_key_iv: "9a59ce54766fc20962a1cae6037f298d"
server_key_version: "encryptionkey@0"
I20260928 16:39:44.096624 952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "c146d4e873164b5e8d65d04f6d9a460d"
format_stamp: "Formatted at 2026-09-28 16:39:44 on dist-test-slave-nqgh"
server_key: "e1a740399df7f85152a51707cfc5d54b"
server_key_iv: "9a59ce54766fc20962a1cae6037f298d"
server_key_version: "encryptionkey@0"
I20260928 16:39:44.101310 952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:39:44.104823 967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:44.106779 952 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:39:44.106982 952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "c146d4e873164b5e8d65d04f6d9a460d"
format_stamp: "Formatted at 2026-09-28 16:39:44 on dist-test-slave-nqgh"
server_key: "e1a740399df7f85152a51707cfc5d54b"
server_key_iv: "9a59ce54766fc20962a1cae6037f298d"
server_key_version: "encryptionkey@0"
I20260928 16:39:44.107112 952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:44.128947 952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:44.132933 952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:44.133147 952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:44.142211 952 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:36649
I20260928 16:39:44.142386 1029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:36649 every 8 connection(s)
I20260928 16:39:44.143429 952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:39:44.147485 1030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:44.148332 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 952
I20260928 16:39:44.148506 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:39:44.148890 25885 external_mini_cluster.cc:1428] Setting key cb8d6a13b7ddd27b788f3d2de5efff61
I20260928 16:39:44.155580 1030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d: Bootstrap starting.
I20260928 16:39:44.158751 1030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d: Neither blocks nor log segments found. Creating new log.
Sep 28 16:39:44 dist-test-slave-nqgh krb5kdc[938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM
I20260928 16:39:44.159825 1030 log.cc:824] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:44.162946 1030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d: No bootstrap required, opened a new log
I20260928 16:39:44.165828 1033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:44.151015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51768 (local address 127.25.71.126:36649)
0928 16:39:44.151646 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:44.151665 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:44.151711 (+ 46us) server_negotiation.cc:407] Connection header received
0928 16:39:44.152610 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:44.152631 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:44.153032 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:44.153377 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:44.154243 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.155354 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.156228 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.156735 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.156901 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:44.159947 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:44.159956 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:44.159980 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:44.160017 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:44.162766 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:44.163271 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:44.163278 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:44.163285 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:44.163410 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:44.163672 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:44.163675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:44.163676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:44.164140 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:44.164327 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:44.164635 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 16:39:44.165009 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":203,"threads_started":1}
I20260928 16:39:44.166996 1030 raft_consensus.cc:348] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c146d4e873164b5e8d65d04f6d9a460d" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 36649 } }
I20260928 16:39:44.167382 1030 raft_consensus.cc:374] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:44.167459 1030 raft_consensus.cc:729] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c146d4e873164b5e8d65d04f6d9a460d, State: Initialized, Role: FOLLOWER
I20260928 16:39:44.168058 1030 consensus_queue.cc:260] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c146d4e873164b5e8d65d04f6d9a460d" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 36649 } }
I20260928 16:39:44.168238 1030 raft_consensus.cc:388] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:39:44.168310 1030 raft_consensus.cc:482] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:39:44.168416 1030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:44.169697 1030 raft_consensus.cc:504] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c146d4e873164b5e8d65d04f6d9a460d" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 36649 } }
I20260928 16:39:44.170148 1030 leader_election.cc:302] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c146d4e873164b5e8d65d04f6d9a460d; no voters:
I20260928 16:39:44.170552 1030 leader_election.cc:288] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:39:44.170861 1035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:44.171167 1035 raft_consensus.cc:686] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d [term 1 LEADER]: Becoming Leader. State: Replica: c146d4e873164b5e8d65d04f6d9a460d, State: Running, Role: LEADER
I20260928 16:39:44.171554 1035 consensus_queue.cc:237] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c146d4e873164b5e8d65d04f6d9a460d" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 36649 } }
I20260928 16:39:44.172259 1030 sys_catalog.cc:564] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:44.173444 1037 sys_catalog.cc:455] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c146d4e873164b5e8d65d04f6d9a460d. Latest consensus state: current_term: 1 leader_uuid: "c146d4e873164b5e8d65d04f6d9a460d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c146d4e873164b5e8d65d04f6d9a460d" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 36649 } } }
I20260928 16:39:44.173630 1037 sys_catalog.cc:458] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:44.173945 1036 sys_catalog.cc:455] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c146d4e873164b5e8d65d04f6d9a460d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c146d4e873164b5e8d65d04f6d9a460d" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 36649 } } }
I20260928 16:39:44.174052 1036 sys_catalog.cc:458] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:44.174394 1044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:39:44.178386 1044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:39:44.185643 1044 catalog_manager.cc:1347] Generated new cluster ID: ded69c52d2924e1d8648186db2559d30
I20260928 16:39:44.185753 1044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:39:44.199672 1044 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:39:44.200762 1044 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:39:44.205731 1044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d: Generated new TSK 0
I20260928 16:39:44.207100 1044 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.25.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.65.keytab.
Entry for principal oryx/127.25.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.65.keytab.
I20260928 16:39:44.247272 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:36649
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.65.keytab
--principal=oryx/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:42927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:44.371460 1056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:44.371768 1056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:44.371842 1056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:44.376302 1056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:44.376386 1056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:44.376428 1056 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:39:44.376461 1056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:39:44.382541 1056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:42927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.65.keytab
--principal=oryx/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.71.126:36649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:44.383857 1056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:44.385126 1056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:44.396669 1062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:44.396669 1064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:44.396726 1061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:44.397094 1056 server_base.cc:1034] running on GCE node
I20260928 16:39:44.397761 1056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:44.398519 1056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:44.399705 1056 hybrid_clock.cc:648] HybridClock initialized: now 1790613584399670 us; error 61 us; skew 500 ppm
Sep 28 16:39:44 dist-test-slave-nqgh krb5kdc[938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613584, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:44.403123 1056 init.cc:377] Logged in from keytab as oryx/127.25.71.65@KRBTEST.COM (short username oryx)
I20260928 16:39:44.404241 1056 webserver.cc:460] Webserver started at http://127.25.71.65:37513/ using document root <none> and password file <none>
I20260928 16:39:44.404902 1056 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:44.404978 1056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:44.405164 1056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:44.407694 1056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "889a7f814a484de2b6ebe5dfa2ec46b2"
format_stamp: "Formatted at 2026-09-28 16:39:44 on dist-test-slave-nqgh"
server_key: "d0244a80034f63a3f4dec5a4fcc4068b"
server_key_iv: "5b10555a8c41053b649a23d366286ee8"
server_key_version: "encryptionkey@0"
I20260928 16:39:44.408347 1056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "889a7f814a484de2b6ebe5dfa2ec46b2"
format_stamp: "Formatted at 2026-09-28 16:39:44 on dist-test-slave-nqgh"
server_key: "d0244a80034f63a3f4dec5a4fcc4068b"
server_key_iv: "5b10555a8c41053b649a23d366286ee8"
server_key_version: "encryptionkey@0"
I20260928 16:39:44.413103 1056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:39:44.416433 1071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:44.418424 1056 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 16:39:44.418594 1056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "889a7f814a484de2b6ebe5dfa2ec46b2"
format_stamp: "Formatted at 2026-09-28 16:39:44 on dist-test-slave-nqgh"
server_key: "d0244a80034f63a3f4dec5a4fcc4068b"
server_key_iv: "5b10555a8c41053b649a23d366286ee8"
server_key_version: "encryptionkey@0"
I20260928 16:39:44.418805 1056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:44.433080 1056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:44.437319 1056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:44.437539 1056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:44.439669 1056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:44.439754 1056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:44.439800 1056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:44.439814 1056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:39:44 dist-test-slave-nqgh krb5kdc[938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613584, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM
I20260928 16:39:44.452406 1056 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:34161
I20260928 16:39:44.452463 1187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:34161 every 8 connection(s)
I20260928 16:39:44.453449 1056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:39:44.456566 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 1056
I20260928 16:39:44.456727 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:39:44.457127 25885 external_mini_cluster.cc:1428] Setting key fa0e60aa29654989def4ef8ed6ee2ca1
I20260928 16:39:44.457706 1033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:44.440968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:46291 (local address 127.25.71.126:36649)
0928 16:39:44.441253 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:44.441260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:44.442439 (+ 1179us) server_negotiation.cc:407] Connection header received
0928 16:39:44.443965 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:44.443970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:44.444151 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:44.444280 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:44.446212 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.447172 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.448769 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.449051 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.449161 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:44.452963 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:44.452967 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:44.452970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:44.453008 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:44.454786 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:44.455458 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:44.455461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:44.455463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:44.455593 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:44.456101 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:44.456105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:44.456106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:44.456365 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:44.456473 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:44.457374 (+ 901us) server_negotiation.cc:299] Negotiation successful
0928 16:39:44.457540 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 16:39:44.459149 1080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:44.441460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:36649 (local address 127.25.71.65:46291)
0928 16:39:44.442166 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:44.442259 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:44.443561 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:44.444541 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:44.444551 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:44.445105 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:44.445921 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:44.445948 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.447366 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.447370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:44.448593 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:44.448612 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.449275 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.449281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:44.449509 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:44.450404 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:44.450434 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:44.452650 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:44.454965 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:44.454971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:44.454988 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:44.455311 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:44.455739 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:44.455742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:44.455744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:44.455947 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:44.456518 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:44.456527 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:44.456993 (+ 466us) client_negotiation.cc:769] Sending connection context
0928 16:39:44.457265 (+ 272us) client_negotiation.cc:240] Negotiation successful
0928 16:39:44.457811 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":152,"threads_started":1}
I20260928 16:39:44.461006 1188 heartbeater.cc:347] Connected to a master server at 127.25.71.126:36649
I20260928 16:39:44.461304 1188 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:44.462081 1188 heartbeater.cc:511] Master 127.25.71.126:36649 requested a full tablet report, sending...
I20260928 16:39:44.464362 983 ts_manager.cc:194] Registered new tserver with Master: 889a7f814a484de2b6ebe5dfa2ec46b2 (127.25.71.65:34161)
I20260928 16:39:44.465767 983 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:46291
WARNING: no policy specified for oryx/127.25.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.71.66@KRBTEST.COM" created.
I20260928 16:39:44.476012 1033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:44.467347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:41457 (local address 127.25.71.126:36649)
0928 16:39:44.467502 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:44.467507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:44.467739 (+ 232us) server_negotiation.cc:407] Connection header received
0928 16:39:44.467883 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:44.467886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:44.467934 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:44.468002 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:44.468551 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.469104 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.469951 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.470271 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.470384 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:44.472065 (+ 1681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:44.472077 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:44.472081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:44.472119 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:44.473971 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:44.474553 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:44.474558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:44.474560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:44.474725 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:44.475036 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:44.475039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:44.475042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:44.475280 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:44.475374 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:44.475668 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 16:39:44.475813 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:39:44.476028 1195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:44.467232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:36649 (local address 127.25.71.65:41457)
0928 16:39:44.467646 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:44.467660 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:44.467788 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:44.468054 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:44.468057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:44.468303 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:44.468435 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:44.468442 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.469237 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.469240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:44.469841 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:44.469849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.470445 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.470450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:44.470585 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:44.471297 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:44.471315 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:44.471838 (+ 523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:44.474166 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:44.474169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:44.474171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:44.474426 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:44.474855 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:44.474858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:44.474860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:44.474949 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:44.475372 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:44.475375 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:44.475496 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 16:39:44.475601 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 16:39:44.475805 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":166,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.66.keytab.
Entry for principal oryx/127.25.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.66.keytab.
I20260928 16:39:44.490417 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:36649
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.66.keytab
--principal=oryx/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:42927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:44.609397 1197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:44.609709 1197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:44.609756 1197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:44.614097 1197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:44.614189 1197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:44.614230 1197 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:39:44.614265 1197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:39:44.619889 1197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:42927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.66.keytab
--principal=oryx/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.71.126:36649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:44.621009 1197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:44.622011 1197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:44.633190 1202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:44.633246 1203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:44.633246 1205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:44.633435 1197 server_base.cc:1034] running on GCE node
I20260928 16:39:44.633991 1197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:44.634796 1197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:44.636273 1197 hybrid_clock.cc:648] HybridClock initialized: now 1790613584636239 us; error 80 us; skew 500 ppm
Sep 28 16:39:44 dist-test-slave-nqgh krb5kdc[938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613584, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:44.640108 1197 init.cc:377] Logged in from keytab as oryx/127.25.71.66@KRBTEST.COM (short username oryx)
I20260928 16:39:44.641363 1197 webserver.cc:460] Webserver started at http://127.25.71.66:40543/ using document root <none> and password file <none>
I20260928 16:39:44.642001 1197 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:44.642076 1197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:44.642282 1197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:44.644620 1197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "e936fc7a2f5c435492b91d9beb57965f"
format_stamp: "Formatted at 2026-09-28 16:39:44 on dist-test-slave-nqgh"
server_key: "17dc86a4b3c50c809677da0fea235638"
server_key_iv: "b73dc9c6335b29cf965cf3b42e80a106"
server_key_version: "encryptionkey@0"
I20260928 16:39:44.645280 1197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "e936fc7a2f5c435492b91d9beb57965f"
format_stamp: "Formatted at 2026-09-28 16:39:44 on dist-test-slave-nqgh"
server_key: "17dc86a4b3c50c809677da0fea235638"
server_key_iv: "b73dc9c6335b29cf965cf3b42e80a106"
server_key_version: "encryptionkey@0"
I20260928 16:39:44.649617 1197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 16:39:44.652849 1212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:44.654541 1197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:39:44.654754 1197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "e936fc7a2f5c435492b91d9beb57965f"
format_stamp: "Formatted at 2026-09-28 16:39:44 on dist-test-slave-nqgh"
server_key: "17dc86a4b3c50c809677da0fea235638"
server_key_iv: "b73dc9c6335b29cf965cf3b42e80a106"
server_key_version: "encryptionkey@0"
I20260928 16:39:44.654963 1197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:44.668394 1197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:44.672307 1197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:44.672531 1197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:44.674786 1197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:44.674862 1197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:44.674919 1197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:44.674939 1197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:44.684922 1197 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:44633
I20260928 16:39:44.685005 1328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:44633 every 8 connection(s)
I20260928 16:39:44.686111 1197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
Sep 28 16:39:44 dist-test-slave-nqgh krb5kdc[938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613584, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM
I20260928 16:39:44.689189 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 1197
I20260928 16:39:44.689366 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:39:44.689724 25885 external_mini_cluster.cc:1428] Setting key 3df6ac8e99ef26aabc5df025c0097c12
I20260928 16:39:44.693032 1033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:44.675915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:58057 (local address 127.25.71.126:36649)
0928 16:39:44.676105 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:44.676111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:44.677292 (+ 1181us) server_negotiation.cc:407] Connection header received
0928 16:39:44.678801 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:44.678806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:44.678891 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:44.679018 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:44.681182 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.682011 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.684093 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.684431 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.684583 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:44.688136 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:44.688142 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:44.688145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:44.688185 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:44.690091 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:44.690883 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:44.690886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:44.690887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:44.690968 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:44.691371 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:44.691377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:44.691379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:44.691731 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:44.691850 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:44.692702 (+ 852us) server_negotiation.cc:299] Negotiation successful
0928 16:39:44.692873 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:39:44.694074 1221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:44.676476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:36649 (local address 127.25.71.66:58057)
0928 16:39:44.677074 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:44.677136 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:44.678387 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:44.679392 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:44.679404 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:44.680038 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:44.680938 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:44.680965 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.682231 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.682234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:44.683894 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:44.683912 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.684815 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.684835 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:44.685089 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:44.685798 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:44.685834 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:44.687885 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:44.690264 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:44.690272 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:44.690290 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:44.690757 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:44.691054 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:44.691058 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:44.691060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:44.691212 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:44.691951 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:44.691959 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:44.692352 (+ 393us) client_negotiation.cc:769] Sending connection context
0928 16:39:44.692582 (+ 230us) client_negotiation.cc:240] Negotiation successful
0928 16:39:44.692995 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":131,"threads_started":1}
I20260928 16:39:44.695670 1329 heartbeater.cc:347] Connected to a master server at 127.25.71.126:36649
I20260928 16:39:44.695899 1329 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:44.696784 1329 heartbeater.cc:511] Master 127.25.71.126:36649 requested a full tablet report, sending...
I20260928 16:39:44.698414 980 ts_manager.cc:194] Registered new tserver with Master: e936fc7a2f5c435492b91d9beb57965f (127.25.71.66:44633)
I20260928 16:39:44.699350 980 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:58057
WARNING: no policy specified for oryx/127.25.71.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.71.67@KRBTEST.COM" created.
I20260928 16:39:44.709954 1033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:44.699124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:53319 (local address 127.25.71.126:36649)
0928 16:39:44.699265 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:44.699270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:44.701335 (+ 2065us) server_negotiation.cc:407] Connection header received
0928 16:39:44.701471 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:44.701474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:44.701537 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:44.701652 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:44.702433 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.703241 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.704374 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.704724 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.704837 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:44.706340 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:44.706343 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:44.706345 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:44.706373 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:44.707943 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:44.708537 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:44.708540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:44.708542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:44.708642 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:44.708954 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:44.708957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:44.708958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:44.709229 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:44.709315 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:44.709671 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 16:39:44.709796 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:39:44.710034 1336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:44.699057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:36649 (local address 127.25.71.66:53319)
0928 16:39:44.701217 (+ 2160us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:44.701232 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:44.701364 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:44.701749 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:44.701753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:44.702134 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:44.702289 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:44.702298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.703395 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.703397 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:44.704214 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:44.704227 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.704889 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.704892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:44.704982 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:44.705632 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:44.705645 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:44.706143 (+ 498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:44.708089 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:44.708092 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:44.708094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:44.708383 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:44.708749 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:44.708752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:44.708754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:44.708857 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:44.709345 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:44.709348 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:44.709489 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 16:39:44.709610 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 16:39:44.709790 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2042,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.67.keytab.
Entry for principal oryx/127.25.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.67.keytab.
I20260928 16:39:44.722564 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:36649
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.67.keytab
--principal=oryx/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:42927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:44.838155 1338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:44.838455 1338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:44.838516 1338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:44.842726 1338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:44.842832 1338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:44.842893 1338 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:39:44.842944 1338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:39:44.848680 1338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:42927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.67.keytab
--principal=oryx/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.71.126:36649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:44.849825 1338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:44.850908 1338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:44.861866 1343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:44.861892 1346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:44.862051 1344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:44.862396 1338 server_base.cc:1034] running on GCE node
I20260928 16:39:44.863024 1338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:44.863924 1338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:44.865150 1338 hybrid_clock.cc:648] HybridClock initialized: now 1790613584865095 us; error 74 us; skew 500 ppm
Sep 28 16:39:44 dist-test-slave-nqgh krb5kdc[938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613584, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:44.868908 1338 init.cc:377] Logged in from keytab as oryx/127.25.71.67@KRBTEST.COM (short username oryx)
I20260928 16:39:44.870282 1338 webserver.cc:460] Webserver started at http://127.25.71.67:44303/ using document root <none> and password file <none>
I20260928 16:39:44.871064 1338 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:44.871155 1338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:44.871405 1338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:44.874361 1338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "7d38c42df27547c49d75484712cc6ef2"
format_stamp: "Formatted at 2026-09-28 16:39:44 on dist-test-slave-nqgh"
server_key: "303286e5add21272d77b39cd368fb40b"
server_key_iv: "3bfd337abc94424e3bdbfee48fd71bb3"
server_key_version: "encryptionkey@0"
I20260928 16:39:44.875239 1338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "7d38c42df27547c49d75484712cc6ef2"
format_stamp: "Formatted at 2026-09-28 16:39:44 on dist-test-slave-nqgh"
server_key: "303286e5add21272d77b39cd368fb40b"
server_key_iv: "3bfd337abc94424e3bdbfee48fd71bb3"
server_key_version: "encryptionkey@0"
I20260928 16:39:44.881210 1338 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:39:44.885142 1354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:44.887250 1338 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:44.887456 1338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "7d38c42df27547c49d75484712cc6ef2"
format_stamp: "Formatted at 2026-09-28 16:39:44 on dist-test-slave-nqgh"
server_key: "303286e5add21272d77b39cd368fb40b"
server_key_iv: "3bfd337abc94424e3bdbfee48fd71bb3"
server_key_version: "encryptionkey@0"
I20260928 16:39:44.887624 1338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:44.912633 1338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:44.916566 1338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:44.916795 1338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:44.918746 1338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:44.918808 1338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:44.918870 1338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:44.918897 1338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:39:44 dist-test-slave-nqgh krb5kdc[938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613584, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM
I20260928 16:39:44.931540 1338 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:38805
I20260928 16:39:44.931773 1470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:38805 every 8 connection(s)
I20260928 16:39:44.932919 1338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:39:44.934693 1033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:44.919881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:56161 (local address 127.25.71.126:36649)
0928 16:39:44.920079 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:44.920085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:44.921028 (+ 943us) server_negotiation.cc:407] Connection header received
0928 16:39:44.922244 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:44.922249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:44.922344 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:44.922491 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:44.924024 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.924916 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.926361 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.926610 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.926748 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:44.930088 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:44.930093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:44.930097 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:44.930136 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:44.931932 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:44.932660 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:44.932663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:44.932665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:44.932746 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:44.933134 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:44.933137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:44.933138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:44.933431 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:44.933534 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:44.934329 (+ 795us) server_negotiation.cc:299] Negotiation successful
0928 16:39:44.934490 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:39:44.935461 1363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:44.920265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:36649 (local address 127.25.71.67:56161)
0928 16:39:44.920848 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:44.920901 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:44.921838 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:44.922716 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:44.922724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:44.923201 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:44.923834 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:44.923853 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.925087 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.925091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:44.926188 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:44.926200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.926812 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.926817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:44.926995 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:44.927720 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:44.927737 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:44.929756 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:44.932073 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:44.932081 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:44.932100 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:44.932522 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:44.932845 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:44.932849 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:44.932851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:44.933044 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:44.933604 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:44.933612 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:44.934022 (+ 410us) client_negotiation.cc:769] Sending connection context
0928 16:39:44.934206 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 16:39:44.934515 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":151,"threads_started":1}
I20260928 16:39:44.937073 1471 heartbeater.cc:347] Connected to a master server at 127.25.71.126:36649
I20260928 16:39:44.937433 1471 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:44.938102 1471 heartbeater.cc:511] Master 127.25.71.126:36649 requested a full tablet report, sending...
I20260928 16:39:44.939546 980 ts_manager.cc:194] Registered new tserver with Master: 7d38c42df27547c49d75484712cc6ef2 (127.25.71.67:38805)
I20260928 16:39:44.940299 980 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:56161
I20260928 16:39:44.941387 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 1338
I20260928 16:39:44.941551 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:39:44.941933 25885 external_mini_cluster.cc:1428] Setting key 1a18accf87f83858fd5113e71ca59e21
I20260928 16:39:44.944844 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:39:44.948802 1033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:44.939841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:48543 (local address 127.25.71.126:36649)
0928 16:39:44.940002 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:44.940007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:44.940212 (+ 205us) server_negotiation.cc:407] Connection header received
0928 16:39:44.940339 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:44.940343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:44.940388 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:44.940458 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:44.941049 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.941699 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.942780 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.943061 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.943160 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:44.944865 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:44.944870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:44.944873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:44.944911 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:44.946751 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:44.947332 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:44.947335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:44.947337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:44.947453 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:44.947847 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:44.947851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:44.947853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:44.948087 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:44.948183 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:44.948463 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 16:39:44.948601 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 16:39:44.948832 1477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:44.939754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:36649 (local address 127.25.71.67:48543)
0928 16:39:44.940129 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:44.940141 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:44.940248 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:44.940581 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:44.940583 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:44.940793 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:44.940915 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:44.940923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.941849 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.941852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:44.942619 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:44.942667 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.943295 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.943298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:44.943376 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:44.943956 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:44.943978 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:44.944477 (+ 499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:44.946959 (+ 2482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:44.946962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:44.946964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:44.947223 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:44.947606 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:44.947611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:44.947613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:44.947730 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:44.948185 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:44.948188 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:44.948300 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 16:39:44.948394 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 16:39:44.948602 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":23,"thread_start_us":113,"threads_started":1}
Sep 28 16:39:44 dist-test-slave-nqgh krb5kdc[938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613584, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:39:44 dist-test-slave-nqgh krb5kdc[938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:44 dist-test-slave-nqgh krb5kdc[938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:39:44.972517 1033 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:44.965650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52166 (local address 127.25.71.126:36649)
0928 16:39:44.965889 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:44.965892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:44.965909 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:39:44.965956 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:44.965959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:44.966002 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:44.966074 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:44.966905 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.967456 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.968311 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.968527 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.968614 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:44.972323 (+ 3709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52166: BlockingRecv error: recv got EOF from 127.0.0.1:52166 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 28 16:39:44 dist-test-slave-nqgh krb5kdc[938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613584, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM
I20260928 16:39:44.985057 1033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:44.974928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.25.71.126:36649)
0928 16:39:44.975098 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:44.975104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:44.975276 (+ 172us) server_negotiation.cc:407] Connection header received
0928 16:39:44.975455 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:44.975459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:44.975532 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:44.975611 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:44.976373 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.977049 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.977924 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:44.978161 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:44.978252 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:44.981223 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:44.981226 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:44.981229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:44.981263 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:44.982975 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:44.983532 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:44.983536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:44.983537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:44.983622 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:44.984062 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:44.984066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:44.984067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:44.984342 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:44.984475 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:44.984736 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 16:39:44.984879 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:39:44.989406 980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:39:44.993664 980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:39:45.012189 1498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.005925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35878 (local address 127.25.71.66:44633)
0928 16:39:45.006423 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:45.006437 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:45.006465 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:39:45.006717 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:45.006724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:45.006965 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:45.007121 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:45.007487 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.008675 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.010576 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.011393 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.011515 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.011702 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:45.011897 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 16:39:45.012015 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":149,"threads_started":1}
I20260928 16:39:45.012358 1495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.005472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:34161 (local address 127.0.0.1:41672)
0928 16:39:45.006417 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:45.006431 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:45.006535 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:45.007401 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:45.007404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:45.007436 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:45.007661 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:45.007670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.008725 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.008728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:45.010456 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:45.010463 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.011809 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.011813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:45.011948 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.011981 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 16:39:45.012069 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 16:39:45.012174 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":162,"spinlock_wait_cycles":3200,"thread_start_us":96,"threads_started":1}
I20260928 16:39:45.012411 1497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.005885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41672 (local address 127.25.71.65:34161)
0928 16:39:45.006703 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:45.006709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:45.006731 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:39:45.006821 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:45.006827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:45.007068 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:45.007212 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:45.007815 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.008601 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.010601 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.011405 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.011535 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.011723 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:45.012126 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 16:39:45.012275 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":113,"threads_started":1}
I20260928 16:39:45.012388 1493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.005183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:44633 (local address 127.0.0.1:35878)
0928 16:39:45.006301 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:45.006317 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:45.006463 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:45.007164 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:45.007170 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:45.007197 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:45.007341 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:45.007347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.008808 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.008811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:45.010428 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:45.010443 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.011541 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.011545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:45.011662 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.011705 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 16:39:45.011834 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 16:39:45.012065 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"mutex_wait_us":73,"thread_start_us":160,"threads_started":1}
I20260928 16:39:45.013072 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.005005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:38805 (local address 127.0.0.1:51580)
0928 16:39:45.006076 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:45.006142 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:45.006676 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:45.007401 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:45.007404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:45.007431 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:45.007667 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:45.007673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.009073 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.009078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:45.010309 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:45.010323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.012626 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.012630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:45.012735 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.012760 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 16:39:45.012845 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 16:39:45.012925 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":75,"spinlock_wait_cycles":1280,"thread_start_us":93,"threads_started":1}
I20260928 16:39:45.013381 1494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.005151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51580 (local address 127.25.71.67:38805)
0928 16:39:45.006131 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:45.006143 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:45.006273 (+ 130us) server_negotiation.cc:407] Connection header received
0928 16:39:45.006828 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:45.006833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:45.007067 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:45.007210 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:45.007819 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.008931 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.010443 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.011393 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.011515 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.011702 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:45.012896 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0928 16:39:45.013063 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":864,"thread_start_us":114,"threads_started":1}
I20260928 16:39:45.016247 1254 tablet_service.cc:1467] Processing CreateTablet for tablet f28e529cd8d54a13917c1d72f89734d5 (DEFAULT_TABLE table=test-table [id=bdd555849f974d0090d1e0556d2a66cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:45.016567 1404 tablet_service.cc:1467] Processing CreateTablet for tablet f28e529cd8d54a13917c1d72f89734d5 (DEFAULT_TABLE table=test-table [id=bdd555849f974d0090d1e0556d2a66cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:45.017445 1254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28e529cd8d54a13917c1d72f89734d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:45.017828 1404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28e529cd8d54a13917c1d72f89734d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:45.019515 1122 tablet_service.cc:1467] Processing CreateTablet for tablet f28e529cd8d54a13917c1d72f89734d5 (DEFAULT_TABLE table=test-table [id=bdd555849f974d0090d1e0556d2a66cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:45.020798 1122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28e529cd8d54a13917c1d72f89734d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:45.027096 1500 tablet_bootstrap.cc:492] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2: Bootstrap starting.
I20260928 16:39:45.028414 1499 tablet_bootstrap.cc:492] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f: Bootstrap starting.
I20260928 16:39:45.028805 1501 tablet_bootstrap.cc:492] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2: Bootstrap starting.
I20260928 16:39:45.030905 1499 tablet_bootstrap.cc:654] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:45.030905 1500 tablet_bootstrap.cc:654] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:45.031050 1501 tablet_bootstrap.cc:654] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:45.031890 1501 log.cc:824] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:45.032011 1499 log.cc:824] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:45.033015 1500 log.cc:824] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:45.034219 1501 tablet_bootstrap.cc:492] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2: No bootstrap required, opened a new log
I20260928 16:39:45.034441 1501 ts_tablet_manager.cc:1397] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:39:45.035121 1499 tablet_bootstrap.cc:492] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f: No bootstrap required, opened a new log
I20260928 16:39:45.035338 1499 ts_tablet_manager.cc:1397] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260928 16:39:45.035589 1500 tablet_bootstrap.cc:492] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2: No bootstrap required, opened a new log
I20260928 16:39:45.035857 1500 ts_tablet_manager.cc:1397] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260928 16:39:45.039484 1501 raft_consensus.cc:348] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } }
I20260928 16:39:45.039484 1499 raft_consensus.cc:348] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } }
I20260928 16:39:45.039841 1501 raft_consensus.cc:374] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:45.039841 1499 raft_consensus.cc:374] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:45.039896 1499 raft_consensus.cc:729] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e936fc7a2f5c435492b91d9beb57965f, State: Initialized, Role: FOLLOWER
I20260928 16:39:45.039896 1501 raft_consensus.cc:729] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889a7f814a484de2b6ebe5dfa2ec46b2, State: Initialized, Role: FOLLOWER
I20260928 16:39:45.040077 1500 raft_consensus.cc:348] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } }
I20260928 16:39:45.040474 1500 raft_consensus.cc:374] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:45.040438 1499 consensus_queue.cc:260] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } }
I20260928 16:39:45.040438 1501 consensus_queue.cc:260] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } }
I20260928 16:39:45.040555 1500 raft_consensus.cc:729] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d38c42df27547c49d75484712cc6ef2, State: Initialized, Role: FOLLOWER
I20260928 16:39:45.041177 1500 consensus_queue.cc:260] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } }
I20260928 16:39:45.041625 1499 ts_tablet_manager.cc:1428] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 16:39:45.041625 1501 ts_tablet_manager.cc:1428] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260928 16:39:45.042047 1329 heartbeater.cc:503] Master 127.25.71.126:36649 was elected leader, sending a full tablet report...
I20260928 16:39:45.042276 1188 heartbeater.cc:503] Master 127.25.71.126:36649 was elected leader, sending a full tablet report...
I20260928 16:39:45.042484 1471 heartbeater.cc:503] Master 127.25.71.126:36649 was elected leader, sending a full tablet report...
I20260928 16:39:45.042547 1500 ts_tablet_manager.cc:1428] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
W20260928 16:39:45.184199 1472 tablet.cc:2369] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:39:45.187799 1330 tablet.cc:2369] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:39:45.205386 1189 tablet.cc:2369] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:39:45.257593 1505 raft_consensus.cc:482] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:39:45.258006 1505 raft_consensus.cc:504] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } }
I20260928 16:39:45.259881 1505 leader_election.cc:288] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d38c42df27547c49d75484712cc6ef2 (127.25.71.67:38805), e936fc7a2f5c435492b91d9beb57965f (127.25.71.66:44633)
I20260928 16:39:45.264357 1508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.260174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:38805 (local address 127.25.71.65:57535)
0928 16:39:45.260597 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:45.260615 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:45.260774 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:45.261038 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:45.261041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:45.261061 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:45.261205 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:45.261211 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.262056 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.262060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:45.263159 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:45.263170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.263840 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.263844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:45.263947 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.263973 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:39:45.264045 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 16:39:45.264146 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":31,"thread_start_us":166,"threads_started":1}
I20260928 16:39:45.264328 1494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.260342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:57535 (local address 127.25.71.67:38805)
0928 16:39:45.260604 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:45.260610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:45.260709 (+ 99us) server_negotiation.cc:407] Connection header received
0928 16:39:45.260895 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:45.260898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:45.260943 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:45.261007 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:45.261304 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.261947 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.263291 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.263712 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.263840 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.263913 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:45.264079 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 16:39:45.264165 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260928 16:39:45.264537 1498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.260372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:55761 (local address 127.25.71.66:44633)
0928 16:39:45.260513 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:45.260519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:45.260825 (+ 306us) server_negotiation.cc:407] Connection header received
0928 16:39:45.261071 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:45.261075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:45.261118 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:45.261178 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:45.261697 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.262285 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.263538 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.263940 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.264060 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.264119 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:45.264311 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 16:39:45.264388 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:39:45.264979 1509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.260322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:44633 (local address 127.25.71.65:55761)
0928 16:39:45.260727 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:45.260742 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:45.260956 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:45.261203 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:45.261206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:45.261223 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:45.261366 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:45.261372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.262409 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.262412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:45.263392 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:45.263403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.264064 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.264068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:45.264192 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.264216 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:39:45.264286 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 16:39:45.264383 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":1,"thread_start_us":148,"threads_started":1}
I20260928 16:39:45.265318 1424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28e529cd8d54a13917c1d72f89734d5" candidate_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d38c42df27547c49d75484712cc6ef2" is_pre_election: true
I20260928 16:39:45.265693 1424 raft_consensus.cc:2454] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 889a7f814a484de2b6ebe5dfa2ec46b2 in term 0.
I20260928 16:39:45.265693 1282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28e529cd8d54a13917c1d72f89734d5" candidate_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e936fc7a2f5c435492b91d9beb57965f" is_pre_election: true
I20260928 16:39:45.265946 1282 raft_consensus.cc:2454] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 889a7f814a484de2b6ebe5dfa2ec46b2 in term 0.
I20260928 16:39:45.266376 1074 leader_election.cc:302] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d38c42df27547c49d75484712cc6ef2, 889a7f814a484de2b6ebe5dfa2ec46b2; no voters:
I20260928 16:39:45.266888 1505 raft_consensus.cc:2781] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:39:45.267007 1505 raft_consensus.cc:482] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:39:45.267038 1505 raft_consensus.cc:3037] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:45.268216 1505 raft_consensus.cc:504] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } }
I20260928 16:39:45.268668 1505 leader_election.cc:288] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 [CANDIDATE]: Term 1 election: Requested vote from peers 7d38c42df27547c49d75484712cc6ef2 (127.25.71.67:38805), e936fc7a2f5c435492b91d9beb57965f (127.25.71.66:44633)
I20260928 16:39:45.269366 1282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28e529cd8d54a13917c1d72f89734d5" candidate_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e936fc7a2f5c435492b91d9beb57965f"
I20260928 16:39:45.269593 1282 raft_consensus.cc:3037] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:45.270922 1424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28e529cd8d54a13917c1d72f89734d5" candidate_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d38c42df27547c49d75484712cc6ef2"
I20260928 16:39:45.271091 1424 raft_consensus.cc:3037] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:45.271246 1282 raft_consensus.cc:2454] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 889a7f814a484de2b6ebe5dfa2ec46b2 in term 1.
I20260928 16:39:45.271801 1072 leader_election.cc:302] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889a7f814a484de2b6ebe5dfa2ec46b2, e936fc7a2f5c435492b91d9beb57965f; no voters:
I20260928 16:39:45.272089 1505 raft_consensus.cc:2781] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:45.272441 1505 raft_consensus.cc:686] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 1 LEADER]: Becoming Leader. State: Replica: 889a7f814a484de2b6ebe5dfa2ec46b2, State: Running, Role: LEADER
I20260928 16:39:45.272884 1505 consensus_queue.cc:237] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } }
I20260928 16:39:45.273149 1424 raft_consensus.cc:2454] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 889a7f814a484de2b6ebe5dfa2ec46b2 in term 1.
I20260928 16:39:45.277060 980 catalog_manager.cc:5495] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 889a7f814a484de2b6ebe5dfa2ec46b2 (127.25.71.65). New cstate: current_term: 1 leader_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } health_report { overall_health: HEALTHY } } }
I20260928 16:39:45.319497 1497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.315164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41680 (local address 127.25.71.65:34161)
0928 16:39:45.315342 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:45.315347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:45.315365 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:39:45.315513 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:45.315516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:45.315581 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:45.315684 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:45.316029 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.316557 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.317546 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.317770 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.317945 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.318258 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:45.318874 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:45.318979 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:45.319235 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 16:39:45.319312 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 16:39:45.327370 1424 raft_consensus.cc:1261] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2 [term 1 FOLLOWER]: Refusing update from remote peer 889a7f814a484de2b6ebe5dfa2ec46b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:45.327370 1282 raft_consensus.cc:1261] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f [term 1 FOLLOWER]: Refusing update from remote peer 889a7f814a484de2b6ebe5dfa2ec46b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:45.328415 1505 consensus_queue.cc:1035] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:45.328696 1510 consensus_queue.cc:1035] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:45.350593 980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:45.350858 980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:45.353937 980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d: Sending DeleteTablet for 3 replicas of tablet f28e529cd8d54a13917c1d72f89734d5
I20260928 16:39:45.355013 1254 tablet_service.cc:1514] Processing DeleteTablet for tablet f28e529cd8d54a13917c1d72f89734d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:45 UTC) from {username='oryx', principal='oryx/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:35878
I20260928 16:39:45.355010 1122 tablet_service.cc:1514] Processing DeleteTablet for tablet f28e529cd8d54a13917c1d72f89734d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:45 UTC) from {username='oryx', principal='oryx/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:41672
I20260928 16:39:45.355222 1404 tablet_service.cc:1514] Processing DeleteTablet for tablet f28e529cd8d54a13917c1d72f89734d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:45 UTC) from {username='oryx', principal='oryx/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:51580
I20260928 16:39:45.355611 1524 tablet_replica.cc:330] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2: stopping tablet replica
I20260928 16:39:45.355768 1523 tablet_replica.cc:330] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f: stopping tablet replica
I20260928 16:39:45.356030 1524 raft_consensus.cc:2229] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:39:45.356037 1523 raft_consensus.cc:2229] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:45.356482 1523 raft_consensus.cc:2258] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:45.356473 1524 raft_consensus.cc:2258] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:45.356923 1525 tablet_replica.cc:330] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2: stopping tablet replica
I20260928 16:39:45.357272 1525 raft_consensus.cc:2229] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:45.357295 980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 16:39:45.357846 1525 raft_consensus.cc:2258] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 16:39:45.358063 980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:39:45.358217 1524 ts_tablet_manager.cc:1905] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:45.360966 1523 ts_tablet_manager.cc:1905] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:45.361327 1525 ts_tablet_manager.cc:1905] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:45.364339 1525 ts_tablet_manager.cc:1918] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:39:45.364530 1525 log.cc:1196] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/f28e529cd8d54a13917c1d72f89734d5
I20260928 16:39:45.364708 1523 ts_tablet_manager.cc:1918] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:39:45.364876 1523 log.cc:1196] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/f28e529cd8d54a13917c1d72f89734d5
I20260928 16:39:45.365235 1523 ts_tablet_manager.cc:1939] T f28e529cd8d54a13917c1d72f89734d5 P e936fc7a2f5c435492b91d9beb57965f: Deleting consensus metadata
I20260928 16:39:45.365525 1122 tablet_service.cc:1467] Processing CreateTablet for tablet 5ecc756824b74d0b97d418a2fa071d46 (DEFAULT_TABLE table=test-table [id=df42761bdef148a9a4a4df6961bb3f39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:45.365525 1254 tablet_service.cc:1467] Processing CreateTablet for tablet 5ecc756824b74d0b97d418a2fa071d46 (DEFAULT_TABLE table=test-table [id=df42761bdef148a9a4a4df6961bb3f39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:45.365823 1122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ecc756824b74d0b97d418a2fa071d46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:45.365823 1254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ecc756824b74d0b97d418a2fa071d46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:45.368145 1404 tablet_service.cc:1467] Processing CreateTablet for tablet 5ecc756824b74d0b97d418a2fa071d46 (DEFAULT_TABLE table=test-table [id=df42761bdef148a9a4a4df6961bb3f39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:45.368482 1501 tablet_bootstrap.cc:492] T 5ecc756824b74d0b97d418a2fa071d46 P 889a7f814a484de2b6ebe5dfa2ec46b2: Bootstrap starting.
I20260928 16:39:45.368546 1404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ecc756824b74d0b97d418a2fa071d46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:45.368826 968 catalog_manager.cc:4841] TS e936fc7a2f5c435492b91d9beb57965f (127.25.71.66:44633): tablet f28e529cd8d54a13917c1d72f89734d5 (table test-table [id=bdd555849f974d0090d1e0556d2a66cf]) successfully deleted
I20260928 16:39:45.369459 1525 ts_tablet_manager.cc:1939] T f28e529cd8d54a13917c1d72f89734d5 P 7d38c42df27547c49d75484712cc6ef2: Deleting consensus metadata
I20260928 16:39:45.369565 1501 tablet_bootstrap.cc:654] T 5ecc756824b74d0b97d418a2fa071d46 P 889a7f814a484de2b6ebe5dfa2ec46b2: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:45.370380 1499 tablet_bootstrap.cc:492] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f: Bootstrap starting.
I20260928 16:39:45.371325 970 catalog_manager.cc:4841] TS 7d38c42df27547c49d75484712cc6ef2 (127.25.71.67:38805): tablet f28e529cd8d54a13917c1d72f89734d5 (table test-table [id=bdd555849f974d0090d1e0556d2a66cf]) successfully deleted
I20260928 16:39:45.371421 1499 tablet_bootstrap.cc:654] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:45.371658 1500 tablet_bootstrap.cc:492] T 5ecc756824b74d0b97d418a2fa071d46 P 7d38c42df27547c49d75484712cc6ef2: Bootstrap starting.
I20260928 16:39:45.372733 1499 tablet_bootstrap.cc:492] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f: No bootstrap required, opened a new log
I20260928 16:39:45.372836 1499 ts_tablet_manager.cc:1397] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:39:45.372972 1500 tablet_bootstrap.cc:654] T 5ecc756824b74d0b97d418a2fa071d46 P 7d38c42df27547c49d75484712cc6ef2: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:45.373660 1501 tablet_bootstrap.cc:492] T 5ecc756824b74d0b97d418a2fa071d46 P 889a7f814a484de2b6ebe5dfa2ec46b2: No bootstrap required, opened a new log
I20260928 16:39:45.373754 1501 ts_tablet_manager.cc:1397] T 5ecc756824b74d0b97d418a2fa071d46 P 889a7f814a484de2b6ebe5dfa2ec46b2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260928 16:39:45.373675 1499 raft_consensus.cc:348] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.374044 1499 raft_consensus.cc:374] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:45.374128 1499 raft_consensus.cc:729] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e936fc7a2f5c435492b91d9beb57965f, State: Initialized, Role: FOLLOWER
I20260928 16:39:45.374300 1501 raft_consensus.cc:348] T 5ecc756824b74d0b97d418a2fa071d46 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.374420 1501 raft_consensus.cc:374] T 5ecc756824b74d0b97d418a2fa071d46 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:45.374455 1501 raft_consensus.cc:729] T 5ecc756824b74d0b97d418a2fa071d46 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889a7f814a484de2b6ebe5dfa2ec46b2, State: Initialized, Role: FOLLOWER
I20260928 16:39:45.374377 1499 consensus_queue.cc:260] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.374567 1501 consensus_queue.cc:260] T 5ecc756824b74d0b97d418a2fa071d46 P 889a7f814a484de2b6ebe5dfa2ec46b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.374867 1501 ts_tablet_manager.cc:1428] T 5ecc756824b74d0b97d418a2fa071d46 P 889a7f814a484de2b6ebe5dfa2ec46b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:39:45.374867 1499 ts_tablet_manager.cc:1428] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 16:39:45.376686 1524 ts_tablet_manager.cc:1918] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:39:45.376838 1524 log.cc:1196] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-0/wal/wals/f28e529cd8d54a13917c1d72f89734d5
I20260928 16:39:45.377645 1524 ts_tablet_manager.cc:1939] T f28e529cd8d54a13917c1d72f89734d5 P 889a7f814a484de2b6ebe5dfa2ec46b2: Deleting consensus metadata
I20260928 16:39:45.379725 1500 tablet_bootstrap.cc:492] T 5ecc756824b74d0b97d418a2fa071d46 P 7d38c42df27547c49d75484712cc6ef2: No bootstrap required, opened a new log
I20260928 16:39:45.379858 1500 ts_tablet_manager.cc:1397] T 5ecc756824b74d0b97d418a2fa071d46 P 7d38c42df27547c49d75484712cc6ef2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260928 16:39:45.380470 1500 raft_consensus.cc:348] T 5ecc756824b74d0b97d418a2fa071d46 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.380628 1500 raft_consensus.cc:374] T 5ecc756824b74d0b97d418a2fa071d46 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:45.380661 1500 raft_consensus.cc:729] T 5ecc756824b74d0b97d418a2fa071d46 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d38c42df27547c49d75484712cc6ef2, State: Initialized, Role: FOLLOWER
I20260928 16:39:45.380836 1500 consensus_queue.cc:260] T 5ecc756824b74d0b97d418a2fa071d46 P 7d38c42df27547c49d75484712cc6ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.381151 1500 ts_tablet_manager.cc:1428] T 5ecc756824b74d0b97d418a2fa071d46 P 7d38c42df27547c49d75484712cc6ef2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 16:39:45.381759 969 catalog_manager.cc:4841] TS 889a7f814a484de2b6ebe5dfa2ec46b2 (127.25.71.65:34161): tablet f28e529cd8d54a13917c1d72f89734d5 (table test-table [id=bdd555849f974d0090d1e0556d2a66cf]) successfully deleted
I20260928 16:39:45.411836 1506 raft_consensus.cc:482] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:39:45.412022 1506 raft_consensus.cc:504] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.413332 1506 leader_election.cc:288] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 889a7f814a484de2b6ebe5dfa2ec46b2 (127.25.71.65:34161), 7d38c42df27547c49d75484712cc6ef2 (127.25.71.67:38805)
I20260928 16:39:45.418109 1497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.413885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:34795 (local address 127.25.71.65:34161)
0928 16:39:45.414030 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:45.414035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:45.414317 (+ 282us) server_negotiation.cc:407] Connection header received
0928 16:39:45.414554 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:45.414557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:45.414600 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:45.414680 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:45.415279 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.415834 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.417002 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.417478 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.417589 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.417648 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:45.417883 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 16:39:45.417968 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 16:39:45.418615 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.413706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:34161 (local address 127.25.71.66:34795)
0928 16:39:45.414224 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:45.414237 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:45.414424 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:45.414839 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:45.414842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:45.414871 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:45.415144 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:45.415153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.415952 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.415954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:45.416873 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:45.416884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.417614 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.417618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:45.417735 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.417759 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:39:45.418359 (+ 600us) client_negotiation.cc:240] Negotiation successful
0928 16:39:45.418454 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":68,"spinlock_wait_cycles":28544,"thread_start_us":171,"threads_started":1}
I20260928 16:39:45.419281 1494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.414047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:38437 (local address 127.25.71.67:38805)
0928 16:39:45.414226 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:45.414232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:45.414323 (+ 91us) server_negotiation.cc:407] Connection header received
0928 16:39:45.414581 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:45.414585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:45.414702 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:45.414817 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:39:45.415275 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.416102 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.417811 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.418396 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.418513 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.418566 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:45.418985 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 16:39:45.419108 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:39:45.419250 1142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ecc756824b74d0b97d418a2fa071d46" candidate_uuid: "e936fc7a2f5c435492b91d9beb57965f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" is_pre_election: true
I20260928 16:39:45.419512 1142 raft_consensus.cc:2454] T 5ecc756824b74d0b97d418a2fa071d46 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e936fc7a2f5c435492b91d9beb57965f in term 0.
I20260928 16:39:45.419802 1530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.413677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:38805 (local address 127.25.71.66:38437)
0928 16:39:45.414203 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:45.414219 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:45.414424 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:45.414839 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:45.414842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:45.414871 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:39:45.415144 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:45.415153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.416313 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.416318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:45.417615 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:45.417631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.418549 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.418553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:45.418809 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.418845 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 16:39:45.419507 (+ 662us) client_negotiation.cc:240] Negotiation successful
0928 16:39:45.419609 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":59,"spinlock_wait_cycles":16000,"thread_start_us":74,"threads_started":1}
I20260928 16:39:45.419992 1214 leader_election.cc:302] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889a7f814a484de2b6ebe5dfa2ec46b2, e936fc7a2f5c435492b91d9beb57965f; no voters:
I20260928 16:39:45.420341 1506 raft_consensus.cc:2781] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:39:45.420408 1506 raft_consensus.cc:482] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:39:45.420444 1506 raft_consensus.cc:3037] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:45.420475 1424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ecc756824b74d0b97d418a2fa071d46" candidate_uuid: "e936fc7a2f5c435492b91d9beb57965f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d38c42df27547c49d75484712cc6ef2" is_pre_election: true
I20260928 16:39:45.420640 1424 raft_consensus.cc:2454] T 5ecc756824b74d0b97d418a2fa071d46 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e936fc7a2f5c435492b91d9beb57965f in term 0.
I20260928 16:39:45.422071 1506 raft_consensus.cc:504] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.422770 1506 leader_election.cc:288] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f [CANDIDATE]: Term 1 election: Requested vote from peers 889a7f814a484de2b6ebe5dfa2ec46b2 (127.25.71.65:34161), 7d38c42df27547c49d75484712cc6ef2 (127.25.71.67:38805)
I20260928 16:39:45.423280 1142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ecc756824b74d0b97d418a2fa071d46" candidate_uuid: "e936fc7a2f5c435492b91d9beb57965f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2"
I20260928 16:39:45.423492 1142 raft_consensus.cc:3037] T 5ecc756824b74d0b97d418a2fa071d46 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:45.425105 1142 raft_consensus.cc:2454] T 5ecc756824b74d0b97d418a2fa071d46 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e936fc7a2f5c435492b91d9beb57965f in term 1.
I20260928 16:39:45.425462 1214 leader_election.cc:302] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889a7f814a484de2b6ebe5dfa2ec46b2, e936fc7a2f5c435492b91d9beb57965f; no voters:
I20260928 16:39:45.425566 1424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ecc756824b74d0b97d418a2fa071d46" candidate_uuid: "e936fc7a2f5c435492b91d9beb57965f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d38c42df27547c49d75484712cc6ef2"
I20260928 16:39:45.425678 1506 raft_consensus.cc:2781] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:45.425791 1424 raft_consensus.cc:3037] T 5ecc756824b74d0b97d418a2fa071d46 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:45.425920 1506 raft_consensus.cc:686] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f [term 1 LEADER]: Becoming Leader. State: Replica: e936fc7a2f5c435492b91d9beb57965f, State: Running, Role: LEADER
I20260928 16:39:45.426383 1506 consensus_queue.cc:237] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.427424 1424 raft_consensus.cc:2454] T 5ecc756824b74d0b97d418a2fa071d46 P 7d38c42df27547c49d75484712cc6ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e936fc7a2f5c435492b91d9beb57965f in term 1.
I20260928 16:39:45.429420 980 catalog_manager.cc:5495] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f reported cstate change: term changed from 0 to 1, leader changed from <none> to e936fc7a2f5c435492b91d9beb57965f (127.25.71.66). New cstate: current_term: 1 leader_uuid: "e936fc7a2f5c435492b91d9beb57965f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } health_report { overall_health: UNKNOWN } } }
Sep 28 16:39:45 dist-test-slave-nqgh krb5kdc[938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613584, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.126@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM
I20260928 16:39:45.484270 1033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.472679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.25.71.126:36649)
0928 16:39:45.472799 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:45.472804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:45.473086 (+ 282us) server_negotiation.cc:407] Connection header received
0928 16:39:45.473231 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:45.473234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:45.473276 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:45.473337 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:45.474056 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.474673 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.475573 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.475966 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.476144 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.479560 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:45.479563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:45.479566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:45.479595 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:45.481484 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:45.482443 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:45.482447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:45.482448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:45.482554 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:45.483016 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:45.483019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:45.483020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:45.483252 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:45.483361 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:45.483942 (+ 581us) server_negotiation.cc:299] Negotiation successful
0928 16:39:45.484093 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 16:39:45.484321 1538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.472646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:36649 (local address 127.0.0.1:52182)
0928 16:39:45.472998 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:45.473010 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:45.473124 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:45.473371 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:45.473373 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:45.473802 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:45.473921 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:45.473929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.474779 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.474781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:45.475367 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:45.475378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.476185 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.476189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:45.476265 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.477060 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:45.477111 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:45.479301 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:45.481684 (+ 2383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:45.481706 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:45.481727 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:45.482249 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:45.482705 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:45.482708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:45.482711 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:45.482871 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:45.483400 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:45.483409 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:45.483601 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 16:39:45.483758 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 16:39:45.484006 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":123,"threads_started":1}
I20260928 16:39:45.489665 980 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:52182:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 16:39:45.490511 980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:39:45.497153 1122 tablet_service.cc:1467] Processing CreateTablet for tablet c85b1b02aa3c4c66b28cff25d25b9170 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=44991ce73ce147bda802b71b2e3c854f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:39:45.497613 1122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c85b1b02aa3c4c66b28cff25d25b9170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:45.497648 1254 tablet_service.cc:1467] Processing CreateTablet for tablet c85b1b02aa3c4c66b28cff25d25b9170 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=44991ce73ce147bda802b71b2e3c854f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:39:45.497993 1254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c85b1b02aa3c4c66b28cff25d25b9170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:45.497812 1404 tablet_service.cc:1467] Processing CreateTablet for tablet c85b1b02aa3c4c66b28cff25d25b9170 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=44991ce73ce147bda802b71b2e3c854f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:39:45.498361 1404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c85b1b02aa3c4c66b28cff25d25b9170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:45.500895 1499 tablet_bootstrap.cc:492] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f: Bootstrap starting.
I20260928 16:39:45.501515 1501 tablet_bootstrap.cc:492] T c85b1b02aa3c4c66b28cff25d25b9170 P 889a7f814a484de2b6ebe5dfa2ec46b2: Bootstrap starting.
I20260928 16:39:45.502097 1499 tablet_bootstrap.cc:654] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:45.502785 1500 tablet_bootstrap.cc:492] T c85b1b02aa3c4c66b28cff25d25b9170 P 7d38c42df27547c49d75484712cc6ef2: Bootstrap starting.
I20260928 16:39:45.503080 1501 tablet_bootstrap.cc:654] T c85b1b02aa3c4c66b28cff25d25b9170 P 889a7f814a484de2b6ebe5dfa2ec46b2: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:45.503983 1499 tablet_bootstrap.cc:492] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f: No bootstrap required, opened a new log
I20260928 16:39:45.504062 1499 ts_tablet_manager.cc:1397] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:39:45.504112 1500 tablet_bootstrap.cc:654] T c85b1b02aa3c4c66b28cff25d25b9170 P 7d38c42df27547c49d75484712cc6ef2: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:45.504464 1499 raft_consensus.cc:348] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.504565 1499 raft_consensus.cc:374] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:45.504585 1499 raft_consensus.cc:729] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e936fc7a2f5c435492b91d9beb57965f, State: Initialized, Role: FOLLOWER
I20260928 16:39:45.504696 1499 consensus_queue.cc:260] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.504977 1499 ts_tablet_manager.cc:1428] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:39:45.505378 1506 tablet_replica.cc:439] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } } }
I20260928 16:39:45.505532 1506 tablet_replica.cc:442] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:39:45.506315 1500 tablet_bootstrap.cc:492] T c85b1b02aa3c4c66b28cff25d25b9170 P 7d38c42df27547c49d75484712cc6ef2: No bootstrap required, opened a new log
I20260928 16:39:45.506448 1500 ts_tablet_manager.cc:1397] T c85b1b02aa3c4c66b28cff25d25b9170 P 7d38c42df27547c49d75484712cc6ef2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 16:39:45.507084 1500 raft_consensus.cc:348] T c85b1b02aa3c4c66b28cff25d25b9170 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.507248 1500 raft_consensus.cc:374] T c85b1b02aa3c4c66b28cff25d25b9170 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:45.507289 1500 raft_consensus.cc:729] T c85b1b02aa3c4c66b28cff25d25b9170 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d38c42df27547c49d75484712cc6ef2, State: Initialized, Role: FOLLOWER
I20260928 16:39:45.507462 1500 consensus_queue.cc:260] T c85b1b02aa3c4c66b28cff25d25b9170 P 7d38c42df27547c49d75484712cc6ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.507822 1500 ts_tablet_manager.cc:1428] T c85b1b02aa3c4c66b28cff25d25b9170 P 7d38c42df27547c49d75484712cc6ef2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 16:39:45.507874 1507 tablet_replica.cc:439] T c85b1b02aa3c4c66b28cff25d25b9170 P 7d38c42df27547c49d75484712cc6ef2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } } }
I20260928 16:39:45.508103 1507 tablet_replica.cc:442] T c85b1b02aa3c4c66b28cff25d25b9170 P 7d38c42df27547c49d75484712cc6ef2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:39:45.510094 1501 tablet_bootstrap.cc:492] T c85b1b02aa3c4c66b28cff25d25b9170 P 889a7f814a484de2b6ebe5dfa2ec46b2: No bootstrap required, opened a new log
I20260928 16:39:45.510223 1501 ts_tablet_manager.cc:1397] T c85b1b02aa3c4c66b28cff25d25b9170 P 889a7f814a484de2b6ebe5dfa2ec46b2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260928 16:39:45.510793 1501 raft_consensus.cc:348] T c85b1b02aa3c4c66b28cff25d25b9170 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.510942 1501 raft_consensus.cc:374] T c85b1b02aa3c4c66b28cff25d25b9170 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:45.510977 1501 raft_consensus.cc:729] T c85b1b02aa3c4c66b28cff25d25b9170 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889a7f814a484de2b6ebe5dfa2ec46b2, State: Initialized, Role: FOLLOWER
I20260928 16:39:45.511107 1501 consensus_queue.cc:260] T c85b1b02aa3c4c66b28cff25d25b9170 P 889a7f814a484de2b6ebe5dfa2ec46b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.511411 1501 ts_tablet_manager.cc:1428] T c85b1b02aa3c4c66b28cff25d25b9170 P 889a7f814a484de2b6ebe5dfa2ec46b2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 16:39:45.511472 1518 tablet_replica.cc:439] T c85b1b02aa3c4c66b28cff25d25b9170 P 889a7f814a484de2b6ebe5dfa2ec46b2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } } }
I20260928 16:39:45.511706 1518 tablet_replica.cc:442] T c85b1b02aa3c4c66b28cff25d25b9170 P 889a7f814a484de2b6ebe5dfa2ec46b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:39:45.572328 1506 raft_consensus.cc:482] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:39:45.572487 1506 raft_consensus.cc:504] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.572923 1506 leader_election.cc:288] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 889a7f814a484de2b6ebe5dfa2ec46b2 (127.25.71.65:34161), 7d38c42df27547c49d75484712cc6ef2 (127.25.71.67:38805)
I20260928 16:39:45.573410 1142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85b1b02aa3c4c66b28cff25d25b9170" candidate_uuid: "e936fc7a2f5c435492b91d9beb57965f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" is_pre_election: true
I20260928 16:39:45.573591 1142 raft_consensus.cc:2454] T c85b1b02aa3c4c66b28cff25d25b9170 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e936fc7a2f5c435492b91d9beb57965f in term 0.
I20260928 16:39:45.573760 1424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85b1b02aa3c4c66b28cff25d25b9170" candidate_uuid: "e936fc7a2f5c435492b91d9beb57965f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d38c42df27547c49d75484712cc6ef2" is_pre_election: true
I20260928 16:39:45.573930 1424 raft_consensus.cc:2454] T c85b1b02aa3c4c66b28cff25d25b9170 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e936fc7a2f5c435492b91d9beb57965f in term 0.
I20260928 16:39:45.574041 1214 leader_election.cc:302] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889a7f814a484de2b6ebe5dfa2ec46b2, e936fc7a2f5c435492b91d9beb57965f; no voters:
I20260928 16:39:45.574411 1506 raft_consensus.cc:2781] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:39:45.574479 1506 raft_consensus.cc:482] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:39:45.574512 1506 raft_consensus.cc:3037] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:45.575804 1506 raft_consensus.cc:504] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.576278 1506 leader_election.cc:288] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f [CANDIDATE]: Term 1 election: Requested vote from peers 889a7f814a484de2b6ebe5dfa2ec46b2 (127.25.71.65:34161), 7d38c42df27547c49d75484712cc6ef2 (127.25.71.67:38805)
I20260928 16:39:45.576781 1142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85b1b02aa3c4c66b28cff25d25b9170" candidate_uuid: "e936fc7a2f5c435492b91d9beb57965f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2"
I20260928 16:39:45.576927 1142 raft_consensus.cc:3037] T c85b1b02aa3c4c66b28cff25d25b9170 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:45.576908 1424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85b1b02aa3c4c66b28cff25d25b9170" candidate_uuid: "e936fc7a2f5c435492b91d9beb57965f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d38c42df27547c49d75484712cc6ef2"
I20260928 16:39:45.577102 1424 raft_consensus.cc:3037] T c85b1b02aa3c4c66b28cff25d25b9170 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:45.578187 1142 raft_consensus.cc:2454] T c85b1b02aa3c4c66b28cff25d25b9170 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e936fc7a2f5c435492b91d9beb57965f in term 1.
I20260928 16:39:45.578262 1424 raft_consensus.cc:2454] T c85b1b02aa3c4c66b28cff25d25b9170 P 7d38c42df27547c49d75484712cc6ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e936fc7a2f5c435492b91d9beb57965f in term 1.
I20260928 16:39:45.578603 1214 leader_election.cc:302] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889a7f814a484de2b6ebe5dfa2ec46b2, e936fc7a2f5c435492b91d9beb57965f; no voters:
I20260928 16:39:45.578886 1506 raft_consensus.cc:2781] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:45.579044 1506 raft_consensus.cc:686] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f [term 1 LEADER]: Becoming Leader. State: Replica: e936fc7a2f5c435492b91d9beb57965f, State: Running, Role: LEADER
I20260928 16:39:45.579209 1506 consensus_queue.cc:237] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.579995 1531 tablet_replica.cc:439] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f: TxnStatusTablet state changed. Reason: New leader e936fc7a2f5c435492b91d9beb57965f. Latest consensus state: current_term: 1 leader_uuid: "e936fc7a2f5c435492b91d9beb57965f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } } }
I20260928 16:39:45.580109 1531 tablet_replica.cc:442] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:39:45.580402 1543 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 16:39:45.580564 1543 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 16:39:45.581588 981 catalog_manager.cc:5495] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f reported cstate change: term changed from 0 to 1, leader changed from <none> to e936fc7a2f5c435492b91d9beb57965f (127.25.71.66). New cstate: current_term: 1 leader_uuid: "e936fc7a2f5c435492b91d9beb57965f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } health_report { overall_health: UNKNOWN } } }
I20260928 16:39:45.600013 1538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.594679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:44633 (local address 127.0.0.1:35894)
0928 16:39:45.594919 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:45.594935 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:45.595074 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:45.595944 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:45.595948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:45.595958 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:39:45.596127 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:45.596135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.597261 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.597264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:45.597964 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:45.597973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.598604 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.598607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:45.598725 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.598795 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:39:45.599679 (+ 884us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:39:45.599694 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 16:39:45.599753 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 16:39:45.599851 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 16:39:45.600025 1498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.595333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35894 (local address 127.25.71.66:44633)
0928 16:39:45.595535 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:45.595540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:45.595562 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:39:45.595657 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:45.595660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:45.595737 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:45.595903 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:45.596309 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.597133 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.598092 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.598451 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.598559 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.598967 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:45.599581 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:45.599696 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:45.599793 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 16:39:45.599862 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 16:39:45.603929 1424 raft_consensus.cc:1261] T c85b1b02aa3c4c66b28cff25d25b9170 P 7d38c42df27547c49d75484712cc6ef2 [term 1 FOLLOWER]: Refusing update from remote peer e936fc7a2f5c435492b91d9beb57965f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:45.604113 1142 raft_consensus.cc:1261] T c85b1b02aa3c4c66b28cff25d25b9170 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 1 FOLLOWER]: Refusing update from remote peer e936fc7a2f5c435492b91d9beb57965f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:45.604638 1531 consensus_queue.cc:1035] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:45.604856 1506 consensus_queue.cc:1035] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f [LEADER]: Connected to new peer: Peer: permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:45.606781 1507 tablet_replica.cc:439] T c85b1b02aa3c4c66b28cff25d25b9170 P 7d38c42df27547c49d75484712cc6ef2: TxnStatusTablet state changed. Reason: New leader e936fc7a2f5c435492b91d9beb57965f. Latest consensus state: current_term: 1 leader_uuid: "e936fc7a2f5c435492b91d9beb57965f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } } }
I20260928 16:39:45.606966 1507 tablet_replica.cc:442] T c85b1b02aa3c4c66b28cff25d25b9170 P 7d38c42df27547c49d75484712cc6ef2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:39:45.606979 1518 tablet_replica.cc:439] T c85b1b02aa3c4c66b28cff25d25b9170 P 889a7f814a484de2b6ebe5dfa2ec46b2: TxnStatusTablet state changed. Reason: New leader e936fc7a2f5c435492b91d9beb57965f. Latest consensus state: current_term: 1 leader_uuid: "e936fc7a2f5c435492b91d9beb57965f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } } }
I20260928 16:39:45.607134 1518 tablet_replica.cc:442] T c85b1b02aa3c4c66b28cff25d25b9170 P 889a7f814a484de2b6ebe5dfa2ec46b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:39:45.609259 1531 tablet_replica.cc:439] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e936fc7a2f5c435492b91d9beb57965f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } } }
I20260928 16:39:45.609431 1531 tablet_replica.cc:442] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:39:45.609537 1531 tablet_replica.cc:439] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e936fc7a2f5c435492b91d9beb57965f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } } }
I20260928 16:39:45.609594 1531 tablet_replica.cc:442] T c85b1b02aa3c4c66b28cff25d25b9170 P e936fc7a2f5c435492b91d9beb57965f: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:39:45.609710 1515 mvcc.cc:205] Tried to move back new op lower bound from 7334353246626766848 to 7334353246533910528. Current Snapshot: MvccSnapshot[applied={T|T < 7334353246626766848 or (T in {7334353246626766848})}]
I20260928 16:39:45.610476 1518 tablet_replica.cc:439] T c85b1b02aa3c4c66b28cff25d25b9170 P 889a7f814a484de2b6ebe5dfa2ec46b2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e936fc7a2f5c435492b91d9beb57965f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } } }
I20260928 16:39:45.610661 1518 tablet_replica.cc:442] T c85b1b02aa3c4c66b28cff25d25b9170 P 889a7f814a484de2b6ebe5dfa2ec46b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:39:45.613125 1507 tablet_replica.cc:439] T c85b1b02aa3c4c66b28cff25d25b9170 P 7d38c42df27547c49d75484712cc6ef2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e936fc7a2f5c435492b91d9beb57965f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } } }
I20260928 16:39:45.613308 1507 tablet_replica.cc:442] T c85b1b02aa3c4c66b28cff25d25b9170 P 7d38c42df27547c49d75484712cc6ef2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:39:45.613947 1514 mvcc.cc:205] Tried to move back new op lower bound from 7334353246626766848 to 7334353246533910528. Current Snapshot: MvccSnapshot[applied={T|T < 7334353246626766848 or (T in {7334353246626766848})}]
I20260928 16:39:45.619760 1498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.616134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35906 (local address 127.25.71.66:44633)
0928 16:39:45.616248 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:45.616254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:45.616309 (+ 55us) server_negotiation.cc:407] Connection header received
0928 16:39:45.616467 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:45.616470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:45.616524 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:45.616886 (+ 362us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:45.616975 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.617522 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.618452 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.618751 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.618852 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.619257 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:45.619342 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:45.619399 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:45.619521 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 16:39:45.619591 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 16:39:45.634987 1498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.630833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:36133 (local address 127.25.71.66:44633)
0928 16:39:45.631001 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:45.631007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:45.631254 (+ 247us) server_negotiation.cc:407] Connection header received
0928 16:39:45.631431 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:45.631435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:45.631520 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:45.631602 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:45.631958 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.632567 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.633552 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.633796 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.633902 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.634248 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:45.634433 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:45.634543 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:45.634689 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 16:39:45.634761 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":71}
I20260928 16:39:45.634994 1551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.630671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:44633 (local address 127.25.71.66:36133)
0928 16:39:45.631127 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:45.631144 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:45.631328 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:45.631610 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:45.631613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:45.631621 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:39:45.631820 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:45.631836 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.632693 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.632697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:45.633420 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:45.633429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.633922 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.633926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:45.634040 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.634106 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:39:45.634564 (+ 458us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:39:45.634570 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:39:45.634682 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 16:39:45.634801 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":5,"thread_start_us":121,"threads_started":1}
I20260928 16:39:45.644805 1424 raft_consensus.cc:1261] T 5ecc756824b74d0b97d418a2fa071d46 P 7d38c42df27547c49d75484712cc6ef2 [term 1 FOLLOWER]: Refusing update from remote peer e936fc7a2f5c435492b91d9beb57965f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:45.644937 1142 raft_consensus.cc:1261] T 5ecc756824b74d0b97d418a2fa071d46 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 1 FOLLOWER]: Refusing update from remote peer e936fc7a2f5c435492b91d9beb57965f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:45.645325 1548 consensus_queue.cc:1035] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:45.645609 1506 consensus_queue.cc:1035] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f [LEADER]: Connected to new peer: Peer: permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:45.701187 980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:45.701406 980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:45.704092 980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d: Sending DeleteTablet for 3 replicas of tablet 5ecc756824b74d0b97d418a2fa071d46
I20260928 16:39:45.704914 1122 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ecc756824b74d0b97d418a2fa071d46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:45 UTC) from {username='oryx', principal='oryx/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:41672
I20260928 16:39:45.704916 1262 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ecc756824b74d0b97d418a2fa071d46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:45 UTC) from {username='oryx', principal='oryx/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:35878
I20260928 16:39:45.705147 1524 tablet_replica.cc:330] T 5ecc756824b74d0b97d418a2fa071d46 P 889a7f814a484de2b6ebe5dfa2ec46b2: stopping tablet replica
I20260928 16:39:45.705147 1523 tablet_replica.cc:330] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f: stopping tablet replica
I20260928 16:39:45.705286 1523 raft_consensus.cc:2229] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:39:45.705283 1524 raft_consensus.cc:2229] T 5ecc756824b74d0b97d418a2fa071d46 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:45.705422 1524 raft_consensus.cc:2258] T 5ecc756824b74d0b97d418a2fa071d46 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:45.705688 1523 raft_consensus.cc:2258] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:45.706669 1524 ts_tablet_manager.cc:1905] T 5ecc756824b74d0b97d418a2fa071d46 P 889a7f814a484de2b6ebe5dfa2ec46b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:45.706707 1523 ts_tablet_manager.cc:1905] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:45.711081 1524 ts_tablet_manager.cc:1918] T 5ecc756824b74d0b97d418a2fa071d46 P 889a7f814a484de2b6ebe5dfa2ec46b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:39:45.711308 1524 log.cc:1196] T 5ecc756824b74d0b97d418a2fa071d46 P 889a7f814a484de2b6ebe5dfa2ec46b2: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-0/wal/wals/5ecc756824b74d0b97d418a2fa071d46
I20260928 16:39:45.711263 1404 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ecc756824b74d0b97d418a2fa071d46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:45 UTC) from {username='oryx', principal='oryx/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:51580
I20260928 16:39:45.711583 1525 tablet_replica.cc:330] T 5ecc756824b74d0b97d418a2fa071d46 P 7d38c42df27547c49d75484712cc6ef2: stopping tablet replica
I20260928 16:39:45.711805 1525 raft_consensus.cc:2229] T 5ecc756824b74d0b97d418a2fa071d46 P 7d38c42df27547c49d75484712cc6ef2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:45.711989 1525 raft_consensus.cc:2258] T 5ecc756824b74d0b97d418a2fa071d46 P 7d38c42df27547c49d75484712cc6ef2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:45.712700 1523 ts_tablet_manager.cc:1918] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:39:45.712800 1523 log.cc:1196] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/5ecc756824b74d0b97d418a2fa071d46
I20260928 16:39:45.713007 1525 ts_tablet_manager.cc:1905] T 5ecc756824b74d0b97d418a2fa071d46 P 7d38c42df27547c49d75484712cc6ef2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:45.713048 1523 ts_tablet_manager.cc:1939] T 5ecc756824b74d0b97d418a2fa071d46 P e936fc7a2f5c435492b91d9beb57965f: Deleting consensus metadata
I20260928 16:39:45.714313 968 catalog_manager.cc:4841] TS e936fc7a2f5c435492b91d9beb57965f (127.25.71.66:44633): tablet 5ecc756824b74d0b97d418a2fa071d46 (table test-table [id=df42761bdef148a9a4a4df6961bb3f39]) successfully deleted
I20260928 16:39:45.716364 1525 ts_tablet_manager.cc:1918] T 5ecc756824b74d0b97d418a2fa071d46 P 7d38c42df27547c49d75484712cc6ef2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:39:45.716454 1525 log.cc:1196] T 5ecc756824b74d0b97d418a2fa071d46 P 7d38c42df27547c49d75484712cc6ef2: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/5ecc756824b74d0b97d418a2fa071d46
I20260928 16:39:45.716696 1525 ts_tablet_manager.cc:1939] T 5ecc756824b74d0b97d418a2fa071d46 P 7d38c42df27547c49d75484712cc6ef2: Deleting consensus metadata
I20260928 16:39:45.717298 1524 ts_tablet_manager.cc:1939] T 5ecc756824b74d0b97d418a2fa071d46 P 889a7f814a484de2b6ebe5dfa2ec46b2: Deleting consensus metadata
I20260928 16:39:45.717730 970 catalog_manager.cc:4841] TS 7d38c42df27547c49d75484712cc6ef2 (127.25.71.67:38805): tablet 5ecc756824b74d0b97d418a2fa071d46 (table test-table [id=df42761bdef148a9a4a4df6961bb3f39]) successfully deleted
I20260928 16:39:45.718304 969 catalog_manager.cc:4841] TS 889a7f814a484de2b6ebe5dfa2ec46b2 (127.25.71.65:34161): tablet 5ecc756824b74d0b97d418a2fa071d46 (table test-table [id=df42761bdef148a9a4a4df6961bb3f39]) successfully deleted
Sep 28 16:39:45 dist-test-slave-nqgh krb5kdc[938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:39:45 dist-test-slave-nqgh krb5kdc[938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:45 dist-test-slave-nqgh krb5kdc[938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:39:45.749089 1033 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:45.740785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.25.71.126:36649)
0928 16:39:45.741003 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:45.741009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:45.741157 (+ 148us) server_negotiation.cc:407] Connection header received
0928 16:39:45.741299 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:45.741303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:45.741371 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:45.741510 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:45.742481 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.743406 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.744549 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.744904 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.745045 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.748931 (+ 3886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52186: BlockingRecv error: recv got EOF from 127.0.0.1:52186 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 16:39:45 dist-test-slave-nqgh krb5kdc[938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM
I20260928 16:39:45.761972 1033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.751119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.25.71.126:36649)
0928 16:39:45.751264 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:45.751268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:45.751295 (+ 27us) server_negotiation.cc:407] Connection header received
0928 16:39:45.751575 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:45.751578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:45.751674 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:45.751772 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:45.752668 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.753265 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.754497 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.754812 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.754910 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.758048 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:45.758053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:45.758056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:45.758100 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:45.759873 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:45.760438 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:45.760442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:45.760445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:45.760534 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:45.760808 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:45.760812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:45.760814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:45.761189 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:45.761331 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:45.761632 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 16:39:45.761778 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:39:45.766994 980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:39:45.767575 980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:39:45.776633 1262 tablet_service.cc:1467] Processing CreateTablet for tablet b93b7ccf3b3d41cda285fcb8a839ce50 (DEFAULT_TABLE table=test-table [id=a1a9a227f93a4e37827de95a0b402007]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:45.776772 1122 tablet_service.cc:1467] Processing CreateTablet for tablet b93b7ccf3b3d41cda285fcb8a839ce50 (DEFAULT_TABLE table=test-table [id=a1a9a227f93a4e37827de95a0b402007]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:45.776959 1404 tablet_service.cc:1467] Processing CreateTablet for tablet b93b7ccf3b3d41cda285fcb8a839ce50 (DEFAULT_TABLE table=test-table [id=a1a9a227f93a4e37827de95a0b402007]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:45.777107 1262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b93b7ccf3b3d41cda285fcb8a839ce50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:45.777165 1122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b93b7ccf3b3d41cda285fcb8a839ce50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:45.777247 1404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b93b7ccf3b3d41cda285fcb8a839ce50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:45.780911 1499 tablet_bootstrap.cc:492] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f: Bootstrap starting.
I20260928 16:39:45.780956 1500 tablet_bootstrap.cc:492] T b93b7ccf3b3d41cda285fcb8a839ce50 P 7d38c42df27547c49d75484712cc6ef2: Bootstrap starting.
I20260928 16:39:45.782121 1499 tablet_bootstrap.cc:654] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:45.782161 1500 tablet_bootstrap.cc:654] T b93b7ccf3b3d41cda285fcb8a839ce50 P 7d38c42df27547c49d75484712cc6ef2: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:45.784022 1500 tablet_bootstrap.cc:492] T b93b7ccf3b3d41cda285fcb8a839ce50 P 7d38c42df27547c49d75484712cc6ef2: No bootstrap required, opened a new log
I20260928 16:39:45.784024 1499 tablet_bootstrap.cc:492] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f: No bootstrap required, opened a new log
I20260928 16:39:45.784135 1499 ts_tablet_manager.cc:1397] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 16:39:45.784140 1500 ts_tablet_manager.cc:1397] T b93b7ccf3b3d41cda285fcb8a839ce50 P 7d38c42df27547c49d75484712cc6ef2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:39:45.784623 1499 raft_consensus.cc:348] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.784662 1500 raft_consensus.cc:348] T b93b7ccf3b3d41cda285fcb8a839ce50 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.784782 1500 raft_consensus.cc:374] T b93b7ccf3b3d41cda285fcb8a839ce50 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:45.784794 1499 raft_consensus.cc:374] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:45.784809 1500 raft_consensus.cc:729] T b93b7ccf3b3d41cda285fcb8a839ce50 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d38c42df27547c49d75484712cc6ef2, State: Initialized, Role: FOLLOWER
I20260928 16:39:45.784833 1499 raft_consensus.cc:729] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e936fc7a2f5c435492b91d9beb57965f, State: Initialized, Role: FOLLOWER
I20260928 16:39:45.784958 1500 consensus_queue.cc:260] T b93b7ccf3b3d41cda285fcb8a839ce50 P 7d38c42df27547c49d75484712cc6ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.785275 1500 ts_tablet_manager.cc:1428] T b93b7ccf3b3d41cda285fcb8a839ce50 P 7d38c42df27547c49d75484712cc6ef2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:39:45.785909 1499 consensus_queue.cc:260] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.786281 1499 ts_tablet_manager.cc:1428] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 16:39:45.786811 1501 tablet_bootstrap.cc:492] T b93b7ccf3b3d41cda285fcb8a839ce50 P 889a7f814a484de2b6ebe5dfa2ec46b2: Bootstrap starting.
I20260928 16:39:45.788301 1501 tablet_bootstrap.cc:654] T b93b7ccf3b3d41cda285fcb8a839ce50 P 889a7f814a484de2b6ebe5dfa2ec46b2: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:45.791697 1501 tablet_bootstrap.cc:492] T b93b7ccf3b3d41cda285fcb8a839ce50 P 889a7f814a484de2b6ebe5dfa2ec46b2: No bootstrap required, opened a new log
I20260928 16:39:45.791836 1501 ts_tablet_manager.cc:1397] T b93b7ccf3b3d41cda285fcb8a839ce50 P 889a7f814a484de2b6ebe5dfa2ec46b2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 16:39:45.792374 1501 raft_consensus.cc:348] T b93b7ccf3b3d41cda285fcb8a839ce50 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.792550 1501 raft_consensus.cc:374] T b93b7ccf3b3d41cda285fcb8a839ce50 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:45.792605 1501 raft_consensus.cc:729] T b93b7ccf3b3d41cda285fcb8a839ce50 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889a7f814a484de2b6ebe5dfa2ec46b2, State: Initialized, Role: FOLLOWER
I20260928 16:39:45.792788 1501 consensus_queue.cc:260] T b93b7ccf3b3d41cda285fcb8a839ce50 P 889a7f814a484de2b6ebe5dfa2ec46b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.793151 1501 ts_tablet_manager.cc:1428] T b93b7ccf3b3d41cda285fcb8a839ce50 P 889a7f814a484de2b6ebe5dfa2ec46b2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 16:39:45.795434 1548 raft_consensus.cc:482] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:39:45.795585 1548 raft_consensus.cc:504] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.796056 1548 leader_election.cc:288] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 889a7f814a484de2b6ebe5dfa2ec46b2 (127.25.71.65:34161), 7d38c42df27547c49d75484712cc6ef2 (127.25.71.67:38805)
I20260928 16:39:45.796614 1142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b93b7ccf3b3d41cda285fcb8a839ce50" candidate_uuid: "e936fc7a2f5c435492b91d9beb57965f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" is_pre_election: true
I20260928 16:39:45.796849 1142 raft_consensus.cc:2454] T b93b7ccf3b3d41cda285fcb8a839ce50 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e936fc7a2f5c435492b91d9beb57965f in term 0.
I20260928 16:39:45.797173 1424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b93b7ccf3b3d41cda285fcb8a839ce50" candidate_uuid: "e936fc7a2f5c435492b91d9beb57965f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d38c42df27547c49d75484712cc6ef2" is_pre_election: true
I20260928 16:39:45.797261 1214 leader_election.cc:302] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889a7f814a484de2b6ebe5dfa2ec46b2, e936fc7a2f5c435492b91d9beb57965f; no voters:
I20260928 16:39:45.797353 1424 raft_consensus.cc:2454] T b93b7ccf3b3d41cda285fcb8a839ce50 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e936fc7a2f5c435492b91d9beb57965f in term 0.
I20260928 16:39:45.797503 1548 raft_consensus.cc:2781] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:39:45.797591 1548 raft_consensus.cc:482] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:39:45.797629 1548 raft_consensus.cc:3037] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:45.799464 1548 raft_consensus.cc:504] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.799952 1548 leader_election.cc:288] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f [CANDIDATE]: Term 1 election: Requested vote from peers 889a7f814a484de2b6ebe5dfa2ec46b2 (127.25.71.65:34161), 7d38c42df27547c49d75484712cc6ef2 (127.25.71.67:38805)
I20260928 16:39:45.800464 1142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b93b7ccf3b3d41cda285fcb8a839ce50" candidate_uuid: "e936fc7a2f5c435492b91d9beb57965f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2"
I20260928 16:39:45.800452 1424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b93b7ccf3b3d41cda285fcb8a839ce50" candidate_uuid: "e936fc7a2f5c435492b91d9beb57965f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d38c42df27547c49d75484712cc6ef2"
I20260928 16:39:45.800633 1142 raft_consensus.cc:3037] T b93b7ccf3b3d41cda285fcb8a839ce50 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:45.800653 1424 raft_consensus.cc:3037] T b93b7ccf3b3d41cda285fcb8a839ce50 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:45.802081 1424 raft_consensus.cc:2454] T b93b7ccf3b3d41cda285fcb8a839ce50 P 7d38c42df27547c49d75484712cc6ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e936fc7a2f5c435492b91d9beb57965f in term 1.
I20260928 16:39:45.802076 1142 raft_consensus.cc:2454] T b93b7ccf3b3d41cda285fcb8a839ce50 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e936fc7a2f5c435492b91d9beb57965f in term 1.
I20260928 16:39:45.802659 1215 leader_election.cc:302] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7d38c42df27547c49d75484712cc6ef2, 889a7f814a484de2b6ebe5dfa2ec46b2, e936fc7a2f5c435492b91d9beb57965f; no voters:
I20260928 16:39:45.802976 1548 raft_consensus.cc:2781] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:45.803176 1548 raft_consensus.cc:686] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f [term 1 LEADER]: Becoming Leader. State: Replica: e936fc7a2f5c435492b91d9beb57965f, State: Running, Role: LEADER
I20260928 16:39:45.803349 1548 consensus_queue.cc:237] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.806414 980 catalog_manager.cc:5495] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f reported cstate change: term changed from 0 to 1, leader changed from <none> to e936fc7a2f5c435492b91d9beb57965f (127.25.71.66). New cstate: current_term: 1 leader_uuid: "e936fc7a2f5c435492b91d9beb57965f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } health_report { overall_health: UNKNOWN } } }
I20260928 16:39:45.818815 1498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:45.815424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35910 (local address 127.25.71.66:44633)
0928 16:39:45.815600 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:45.815615 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:45.815631 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:39:45.815693 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:45.815696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:45.815749 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:45.815812 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:45.816071 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.816674 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.817621 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:45.817852 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:45.817944 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:45.818253 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:45.818352 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:45.818410 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:45.818508 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 16:39:45.818575 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 16:39:45.821740 1424 raft_consensus.cc:1261] T b93b7ccf3b3d41cda285fcb8a839ce50 P 7d38c42df27547c49d75484712cc6ef2 [term 1 FOLLOWER]: Refusing update from remote peer e936fc7a2f5c435492b91d9beb57965f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:45.821743 1142 raft_consensus.cc:1261] T b93b7ccf3b3d41cda285fcb8a839ce50 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 1 FOLLOWER]: Refusing update from remote peer e936fc7a2f5c435492b91d9beb57965f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:45.822346 1506 consensus_queue.cc:1035] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:45.822584 1548 consensus_queue.cc:1035] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f [LEADER]: Connected to new peer: Peer: permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:45.835805 980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:45.836123 980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:45.838953 980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d: Sending DeleteTablet for 3 replicas of tablet b93b7ccf3b3d41cda285fcb8a839ce50
I20260928 16:39:45.840049 1262 tablet_service.cc:1514] Processing DeleteTablet for tablet b93b7ccf3b3d41cda285fcb8a839ce50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:45 UTC) from {username='oryx', principal='oryx/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:35878
I20260928 16:39:45.840242 1122 tablet_service.cc:1514] Processing DeleteTablet for tablet b93b7ccf3b3d41cda285fcb8a839ce50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:45 UTC) from {username='oryx', principal='oryx/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:41672
I20260928 16:39:45.840317 1523 tablet_replica.cc:330] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f: stopping tablet replica
I20260928 16:39:45.840472 1523 raft_consensus.cc:2229] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:39:45.840603 1524 tablet_replica.cc:330] T b93b7ccf3b3d41cda285fcb8a839ce50 P 889a7f814a484de2b6ebe5dfa2ec46b2: stopping tablet replica
I20260928 16:39:45.840760 1523 raft_consensus.cc:2258] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:45.840808 1524 raft_consensus.cc:2229] T b93b7ccf3b3d41cda285fcb8a839ce50 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:45.841122 1524 raft_consensus.cc:2258] T b93b7ccf3b3d41cda285fcb8a839ce50 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:45.841678 1523 ts_tablet_manager.cc:1905] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:45.842270 1404 tablet_service.cc:1514] Processing DeleteTablet for tablet b93b7ccf3b3d41cda285fcb8a839ce50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:45 UTC) from {username='oryx', principal='oryx/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:51580
I20260928 16:39:45.842227 980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 16:39:45.842482 1525 tablet_replica.cc:330] T b93b7ccf3b3d41cda285fcb8a839ce50 P 7d38c42df27547c49d75484712cc6ef2: stopping tablet replica
I20260928 16:39:45.842602 1525 raft_consensus.cc:2229] T b93b7ccf3b3d41cda285fcb8a839ce50 P 7d38c42df27547c49d75484712cc6ef2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:45.842814 1525 raft_consensus.cc:2258] T b93b7ccf3b3d41cda285fcb8a839ce50 P 7d38c42df27547c49d75484712cc6ef2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 16:39:45.842855 980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:39:45.844425 1525 ts_tablet_manager.cc:1905] T b93b7ccf3b3d41cda285fcb8a839ce50 P 7d38c42df27547c49d75484712cc6ef2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:45.844986 1523 ts_tablet_manager.cc:1918] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:39:45.845072 1523 log.cc:1196] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/b93b7ccf3b3d41cda285fcb8a839ce50
I20260928 16:39:45.845259 1523 ts_tablet_manager.cc:1939] T b93b7ccf3b3d41cda285fcb8a839ce50 P e936fc7a2f5c435492b91d9beb57965f: Deleting consensus metadata
I20260928 16:39:45.846416 968 catalog_manager.cc:4841] TS e936fc7a2f5c435492b91d9beb57965f (127.25.71.66:44633): tablet b93b7ccf3b3d41cda285fcb8a839ce50 (table test-table [id=a1a9a227f93a4e37827de95a0b402007]) successfully deleted
I20260928 16:39:45.846825 1524 ts_tablet_manager.cc:1905] T b93b7ccf3b3d41cda285fcb8a839ce50 P 889a7f814a484de2b6ebe5dfa2ec46b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:45.847600 1525 ts_tablet_manager.cc:1918] T b93b7ccf3b3d41cda285fcb8a839ce50 P 7d38c42df27547c49d75484712cc6ef2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:39:45.847690 1525 log.cc:1196] T b93b7ccf3b3d41cda285fcb8a839ce50 P 7d38c42df27547c49d75484712cc6ef2: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/b93b7ccf3b3d41cda285fcb8a839ce50
I20260928 16:39:45.847980 1525 ts_tablet_manager.cc:1939] T b93b7ccf3b3d41cda285fcb8a839ce50 P 7d38c42df27547c49d75484712cc6ef2: Deleting consensus metadata
I20260928 16:39:45.848966 970 catalog_manager.cc:4841] TS 7d38c42df27547c49d75484712cc6ef2 (127.25.71.67:38805): tablet b93b7ccf3b3d41cda285fcb8a839ce50 (table test-table [id=a1a9a227f93a4e37827de95a0b402007]) successfully deleted
I20260928 16:39:45.849714 1524 ts_tablet_manager.cc:1918] T b93b7ccf3b3d41cda285fcb8a839ce50 P 889a7f814a484de2b6ebe5dfa2ec46b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:39:45.849814 1524 log.cc:1196] T b93b7ccf3b3d41cda285fcb8a839ce50 P 889a7f814a484de2b6ebe5dfa2ec46b2: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-0/wal/wals/b93b7ccf3b3d41cda285fcb8a839ce50
I20260928 16:39:45.850075 1524 ts_tablet_manager.cc:1939] T b93b7ccf3b3d41cda285fcb8a839ce50 P 889a7f814a484de2b6ebe5dfa2ec46b2: Deleting consensus metadata
I20260928 16:39:45.851239 969 catalog_manager.cc:4841] TS 889a7f814a484de2b6ebe5dfa2ec46b2 (127.25.71.65:34161): tablet b93b7ccf3b3d41cda285fcb8a839ce50 (table test-table [id=a1a9a227f93a4e37827de95a0b402007]) successfully deleted
I20260928 16:39:45.854027 1262 tablet_service.cc:1467] Processing CreateTablet for tablet 8fab915e500b4750b307bdbdeb51da00 (DEFAULT_TABLE table=test-table [id=98944711580f4212939d47aad43b809c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:45.854214 1122 tablet_service.cc:1467] Processing CreateTablet for tablet 8fab915e500b4750b307bdbdeb51da00 (DEFAULT_TABLE table=test-table [id=98944711580f4212939d47aad43b809c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:45.854214 1404 tablet_service.cc:1467] Processing CreateTablet for tablet 8fab915e500b4750b307bdbdeb51da00 (DEFAULT_TABLE table=test-table [id=98944711580f4212939d47aad43b809c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:39:45.854390 1262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fab915e500b4750b307bdbdeb51da00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:45.854591 1122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fab915e500b4750b307bdbdeb51da00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:45.854601 1404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fab915e500b4750b307bdbdeb51da00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:45.857923 1500 tablet_bootstrap.cc:492] T 8fab915e500b4750b307bdbdeb51da00 P 7d38c42df27547c49d75484712cc6ef2: Bootstrap starting.
I20260928 16:39:45.857935 1501 tablet_bootstrap.cc:492] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2: Bootstrap starting.
I20260928 16:39:45.858438 1499 tablet_bootstrap.cc:492] T 8fab915e500b4750b307bdbdeb51da00 P e936fc7a2f5c435492b91d9beb57965f: Bootstrap starting.
I20260928 16:39:45.859263 1501 tablet_bootstrap.cc:654] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:45.859648 1500 tablet_bootstrap.cc:654] T 8fab915e500b4750b307bdbdeb51da00 P 7d38c42df27547c49d75484712cc6ef2: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:45.859781 1499 tablet_bootstrap.cc:654] T 8fab915e500b4750b307bdbdeb51da00 P e936fc7a2f5c435492b91d9beb57965f: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:45.861017 1499 tablet_bootstrap.cc:492] T 8fab915e500b4750b307bdbdeb51da00 P e936fc7a2f5c435492b91d9beb57965f: No bootstrap required, opened a new log
I20260928 16:39:45.861114 1499 ts_tablet_manager.cc:1397] T 8fab915e500b4750b307bdbdeb51da00 P e936fc7a2f5c435492b91d9beb57965f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:39:45.861574 1499 raft_consensus.cc:348] T 8fab915e500b4750b307bdbdeb51da00 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.861693 1499 raft_consensus.cc:374] T 8fab915e500b4750b307bdbdeb51da00 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:45.861727 1499 raft_consensus.cc:729] T 8fab915e500b4750b307bdbdeb51da00 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e936fc7a2f5c435492b91d9beb57965f, State: Initialized, Role: FOLLOWER
I20260928 16:39:45.861725 1501 tablet_bootstrap.cc:492] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2: No bootstrap required, opened a new log
I20260928 16:39:45.861840 1501 ts_tablet_manager.cc:1397] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260928 16:39:45.861871 1499 consensus_queue.cc:260] T 8fab915e500b4750b307bdbdeb51da00 P e936fc7a2f5c435492b91d9beb57965f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.862030 1500 tablet_bootstrap.cc:492] T 8fab915e500b4750b307bdbdeb51da00 P 7d38c42df27547c49d75484712cc6ef2: No bootstrap required, opened a new log
I20260928 16:39:45.862136 1500 ts_tablet_manager.cc:1397] T 8fab915e500b4750b307bdbdeb51da00 P 7d38c42df27547c49d75484712cc6ef2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:45.862143 1499 ts_tablet_manager.cc:1428] T 8fab915e500b4750b307bdbdeb51da00 P e936fc7a2f5c435492b91d9beb57965f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:39:45.862430 1501 raft_consensus.cc:348] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.862597 1501 raft_consensus.cc:374] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:45.862680 1500 raft_consensus.cc:348] T 8fab915e500b4750b307bdbdeb51da00 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.862790 1501 raft_consensus.cc:729] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889a7f814a484de2b6ebe5dfa2ec46b2, State: Initialized, Role: FOLLOWER
I20260928 16:39:45.862838 1500 raft_consensus.cc:374] T 8fab915e500b4750b307bdbdeb51da00 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:45.862874 1500 raft_consensus.cc:729] T 8fab915e500b4750b307bdbdeb51da00 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d38c42df27547c49d75484712cc6ef2, State: Initialized, Role: FOLLOWER
I20260928 16:39:45.863020 1500 consensus_queue.cc:260] T 8fab915e500b4750b307bdbdeb51da00 P 7d38c42df27547c49d75484712cc6ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.863351 1500 ts_tablet_manager.cc:1428] T 8fab915e500b4750b307bdbdeb51da00 P 7d38c42df27547c49d75484712cc6ef2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:39:45.863638 1501 consensus_queue.cc:260] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.865015 1501 ts_tablet_manager.cc:1428] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260928 16:39:45.973094 1518 raft_consensus.cc:482] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:39:45.973268 1518 raft_consensus.cc:504] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.973767 1518 leader_election.cc:288] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e936fc7a2f5c435492b91d9beb57965f (127.25.71.66:44633), 7d38c42df27547c49d75484712cc6ef2 (127.25.71.67:38805)
I20260928 16:39:45.974548 1282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fab915e500b4750b307bdbdeb51da00" candidate_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e936fc7a2f5c435492b91d9beb57965f" is_pre_election: true
I20260928 16:39:45.974548 1424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fab915e500b4750b307bdbdeb51da00" candidate_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d38c42df27547c49d75484712cc6ef2" is_pre_election: true
I20260928 16:39:45.974825 1424 raft_consensus.cc:2454] T 8fab915e500b4750b307bdbdeb51da00 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 889a7f814a484de2b6ebe5dfa2ec46b2 in term 0.
I20260928 16:39:45.974825 1282 raft_consensus.cc:2454] T 8fab915e500b4750b307bdbdeb51da00 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 889a7f814a484de2b6ebe5dfa2ec46b2 in term 0.
I20260928 16:39:45.975292 1074 leader_election.cc:302] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d38c42df27547c49d75484712cc6ef2, 889a7f814a484de2b6ebe5dfa2ec46b2; no voters:
I20260928 16:39:45.975556 1518 raft_consensus.cc:2781] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:39:45.975631 1518 raft_consensus.cc:482] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:39:45.975657 1518 raft_consensus.cc:3037] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:45.976877 1518 raft_consensus.cc:504] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.977288 1518 leader_election.cc:288] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 [CANDIDATE]: Term 1 election: Requested vote from peers e936fc7a2f5c435492b91d9beb57965f (127.25.71.66:44633), 7d38c42df27547c49d75484712cc6ef2 (127.25.71.67:38805)
I20260928 16:39:45.977977 1424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fab915e500b4750b307bdbdeb51da00" candidate_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d38c42df27547c49d75484712cc6ef2"
I20260928 16:39:45.977977 1282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fab915e500b4750b307bdbdeb51da00" candidate_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e936fc7a2f5c435492b91d9beb57965f"
I20260928 16:39:45.978191 1424 raft_consensus.cc:3037] T 8fab915e500b4750b307bdbdeb51da00 P 7d38c42df27547c49d75484712cc6ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:45.978191 1282 raft_consensus.cc:3037] T 8fab915e500b4750b307bdbdeb51da00 P e936fc7a2f5c435492b91d9beb57965f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:45.979812 1424 raft_consensus.cc:2454] T 8fab915e500b4750b307bdbdeb51da00 P 7d38c42df27547c49d75484712cc6ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 889a7f814a484de2b6ebe5dfa2ec46b2 in term 1.
I20260928 16:39:45.979815 1282 raft_consensus.cc:2454] T 8fab915e500b4750b307bdbdeb51da00 P e936fc7a2f5c435492b91d9beb57965f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 889a7f814a484de2b6ebe5dfa2ec46b2 in term 1.
I20260928 16:39:45.980347 1074 leader_election.cc:302] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d38c42df27547c49d75484712cc6ef2, 889a7f814a484de2b6ebe5dfa2ec46b2; no voters:
I20260928 16:39:45.980619 1518 raft_consensus.cc:2781] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:45.980916 1518 raft_consensus.cc:686] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 1 LEADER]: Becoming Leader. State: Replica: 889a7f814a484de2b6ebe5dfa2ec46b2, State: Running, Role: LEADER
I20260928 16:39:45.981114 1518 consensus_queue.cc:237] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } }
I20260928 16:39:45.983475 980 catalog_manager.cc:5495] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 889a7f814a484de2b6ebe5dfa2ec46b2 (127.25.71.65). New cstate: current_term: 1 leader_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889a7f814a484de2b6ebe5dfa2ec46b2" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 } health_report { overall_health: UNKNOWN } } }
I20260928 16:39:46.012914 1579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:46.009050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.25.71.65:34161)
0928 16:39:46.009419 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:46.009424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:46.009442 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:39:46.009507 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:46.009510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:46.009595 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:46.009690 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:46.010030 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:46.010728 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:46.011757 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:46.012029 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:46.012145 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:46.012405 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:46.012504 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:46.012564 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:46.012673 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 16:39:46.012742 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":131,"threads_started":1}
I20260928 16:39:46.026696 1582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:46.022585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:44633 (local address 127.25.71.65:44717)
0928 16:39:46.023001 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:46.023015 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:46.023154 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:46.023508 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:46.023511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:46.023519 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:39:46.023667 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:46.023672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:46.024535 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:46.024538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:46.025242 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:46.025251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:46.025866 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:46.025869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:46.025950 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:46.026010 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:39:46.026327 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:39:46.026331 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 16:39:46.026420 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 16:39:46.026498 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":141,"threads_started":1}
I20260928 16:39:46.026734 1498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:46.022793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:44717 (local address 127.25.71.66:44633)
0928 16:39:46.022947 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:46.022951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:46.023132 (+ 181us) server_negotiation.cc:407] Connection header received
0928 16:39:46.023275 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:46.023278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:46.023336 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:46.023417 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:46.023792 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:46.024382 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:46.025410 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:46.025725 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:46.025847 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:46.026153 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:46.026250 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:46.026315 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:46.026430 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 16:39:46.026523 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:39:46.034790 1424 raft_consensus.cc:1261] T 8fab915e500b4750b307bdbdeb51da00 P 7d38c42df27547c49d75484712cc6ef2 [term 1 FOLLOWER]: Refusing update from remote peer 889a7f814a484de2b6ebe5dfa2ec46b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:46.034883 1282 raft_consensus.cc:1261] T 8fab915e500b4750b307bdbdeb51da00 P e936fc7a2f5c435492b91d9beb57965f [term 1 FOLLOWER]: Refusing update from remote peer 889a7f814a484de2b6ebe5dfa2ec46b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:46.035384 1577 consensus_queue.cc:1035] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d38c42df27547c49d75484712cc6ef2" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 38805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:46.035633 1518 consensus_queue.cc:1035] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e936fc7a2f5c435492b91d9beb57965f" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 44633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:46.054761 1551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:46.051317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:34161 (local address 127.25.71.66:51001)
0928 16:39:46.051575 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:46.051591 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:46.051675 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:46.052143 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:46.052146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:46.052154 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:39:46.052260 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:46.052266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:46.053000 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:46.053003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:46.053608 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:46.053616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:46.054059 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:46.054063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:46.054145 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:46.054179 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:39:46.054439 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:39:46.054445 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:39:46.054505 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 16:39:46.054598 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260928 16:39:46.054761 1579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:46.051807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:51001 (local address 127.25.71.65:34161)
0928 16:39:46.051928 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:46.051933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:46.051946 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:39:46.051993 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:46.051996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:46.052050 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:46.052123 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:46.052360 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:46.052895 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:46.053721 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:46.053958 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:46.054046 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:46.054285 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:46.054367 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:46.054422 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:46.054513 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 16:39:46.054574 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:39:46.082890 984 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:46.083132 984 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:39:46.086114 984 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c146d4e873164b5e8d65d04f6d9a460d: Sending DeleteTablet for 3 replicas of tablet 8fab915e500b4750b307bdbdeb51da00
I20260928 16:39:46.087074 1122 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fab915e500b4750b307bdbdeb51da00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:46 UTC) from {username='oryx', principal='oryx/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:41672
I20260928 16:39:46.087127 1404 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fab915e500b4750b307bdbdeb51da00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:46 UTC) from {username='oryx', principal='oryx/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:51580
I20260928 16:39:46.087091 1262 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fab915e500b4750b307bdbdeb51da00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:39:46 UTC) from {username='oryx', principal='oryx/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:35878
I20260928 16:39:46.087352 1525 tablet_replica.cc:330] T 8fab915e500b4750b307bdbdeb51da00 P 7d38c42df27547c49d75484712cc6ef2: stopping tablet replica
I20260928 16:39:46.087352 1523 tablet_replica.cc:330] T 8fab915e500b4750b307bdbdeb51da00 P e936fc7a2f5c435492b91d9beb57965f: stopping tablet replica
I20260928 16:39:46.087373 1524 tablet_replica.cc:330] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2: stopping tablet replica
I20260928 16:39:46.087500 1523 raft_consensus.cc:2229] T 8fab915e500b4750b307bdbdeb51da00 P e936fc7a2f5c435492b91d9beb57965f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:46.087496 1525 raft_consensus.cc:2229] T 8fab915e500b4750b307bdbdeb51da00 P 7d38c42df27547c49d75484712cc6ef2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:39:46.087520 1524 raft_consensus.cc:2229] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:39:46.087667 1523 raft_consensus.cc:2258] T 8fab915e500b4750b307bdbdeb51da00 P e936fc7a2f5c435492b91d9beb57965f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:46.087663 1525 raft_consensus.cc:2258] T 8fab915e500b4750b307bdbdeb51da00 P 7d38c42df27547c49d75484712cc6ef2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:46.087734 1524 raft_consensus.cc:2258] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:39:46.088645 1525 ts_tablet_manager.cc:1905] T 8fab915e500b4750b307bdbdeb51da00 P 7d38c42df27547c49d75484712cc6ef2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:46.088681 1523 ts_tablet_manager.cc:1905] T 8fab915e500b4750b307bdbdeb51da00 P e936fc7a2f5c435492b91d9beb57965f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:46.088845 1524 ts_tablet_manager.cc:1905] T 8fab915e500b4750b307bdbdeb51da00 P 889a7f814a484de2b6ebe5dfa2ec46b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:39:46.089213 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 1056
I20260928 16:39:46.092176 1523 ts_tablet_manager.cc:1918] T 8fab915e500b4750b307bdbdeb51da00 P e936fc7a2f5c435492b91d9beb57965f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:39:46.092299 1523 log.cc:1196] T 8fab915e500b4750b307bdbdeb51da00 P e936fc7a2f5c435492b91d9beb57965f: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/8fab915e500b4750b307bdbdeb51da00
I20260928 16:39:46.092559 1523 ts_tablet_manager.cc:1939] T 8fab915e500b4750b307bdbdeb51da00 P e936fc7a2f5c435492b91d9beb57965f: Deleting consensus metadata
I20260928 16:39:46.093214 1525 ts_tablet_manager.cc:1918] T 8fab915e500b4750b307bdbdeb51da00 P 7d38c42df27547c49d75484712cc6ef2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:39:46.093319 1525 log.cc:1196] T 8fab915e500b4750b307bdbdeb51da00 P 7d38c42df27547c49d75484712cc6ef2: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/8fab915e500b4750b307bdbdeb51da00
I20260928 16:39:46.093567 1525 ts_tablet_manager.cc:1939] T 8fab915e500b4750b307bdbdeb51da00 P 7d38c42df27547c49d75484712cc6ef2: Deleting consensus metadata
I20260928 16:39:46.093941 968 catalog_manager.cc:4841] TS e936fc7a2f5c435492b91d9beb57965f (127.25.71.66:44633): tablet 8fab915e500b4750b307bdbdeb51da00 (table test-table [id=98944711580f4212939d47aad43b809c]) successfully deleted
I20260928 16:39:46.094453 970 catalog_manager.cc:4841] TS 7d38c42df27547c49d75484712cc6ef2 (127.25.71.67:38805): tablet 8fab915e500b4750b307bdbdeb51da00 (table test-table [id=98944711580f4212939d47aad43b809c]) successfully deleted
W20260928 16:39:46.099113 969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.71.65:34161 (error 108)
I20260928 16:39:46.099433 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 1197
I20260928 16:39:46.103098 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:46.102481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:34161 (local address 127.0.0.1:41706)
0928 16:39:46.102880 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:46.102983 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.65:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":91,"threads_started":1}
W20260928 16:39:46.103329 969 catalog_manager.cc:4584] TS 889a7f814a484de2b6ebe5dfa2ec46b2 (127.25.71.65:34161): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8fab915e500b4750b307bdbdeb51da00: Network error: Client connection negotiation failed: client connection to 127.25.71.65:34161: connect: Connection refused (error 111)
I20260928 16:39:46.109848 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 1338
I20260928 16:39:46.116582 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 952
2026-09-28T16:39:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3322 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:39:46 dist-test-slave-nqgh krb5kdc[1591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:39:46 dist-test-slave-nqgh krb5kdc[1591](info): set up 2 sockets
Sep 28 16:39:46 dist-test-slave-nqgh krb5kdc[1591](info): commencing operation
krb5kdc: starting...
W20260928 16:39:47.085064 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:39:47 dist-test-slave-nqgh krb5kdc[1591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:39:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:39:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:39:47.236757 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:45941
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:45941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:47.372937 1604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:47.373379 1604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:47.373430 1604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:47.378170 1604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:47.378290 1604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:47.378311 1604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:47.378328 1604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:47.378343 1604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:39:47.385427 1604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:45941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:45941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:47.387074 1604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:47.388478 1604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:47.399310 1612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:47.399356 1610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:47.399312 1609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:47.399820 1604 server_base.cc:1034] running on GCE node
I20260928 16:39:47.400625 1604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:47.402155 1604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:47.403373 1604 hybrid_clock.cc:648] HybridClock initialized: now 1790613587403328 us; error 77 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T16:39:47Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1300 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:39:47 dist-test-slave-nqgh krb5kdc[1617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:39:47 dist-test-slave-nqgh krb5kdc[1617](info): set up 2 sockets
Sep 28 16:39:47 dist-test-slave-nqgh krb5kdc[1617](info): commencing operation
krb5kdc: starting...
W20260928 16:39:48.401273 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:39:48 dist-test-slave-nqgh krb5kdc[1617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:39:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:39:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:39:48.551901 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:44945
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:44945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:48.676388 1630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:48.676733 1630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:48.676792 1630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:48.681151 1630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:48.681245 1630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:48.681274 1630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:48.681295 1630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:48.681314 1630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:39:48.687196 1630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:44945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:44945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:48.688417 1630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:48.689534 1630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:48.699440 1635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:48.699440 1636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:48.699416 1638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:48.699806 1630 server_base.cc:1034] running on GCE node
I20260928 16:39:48.700788 1630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:48.702474 1630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:48.703701 1630 hybrid_clock.cc:648] HybridClock initialized: now 1790613588703679 us; error 61 us; skew 500 ppm
Sep 28 16:39:48 dist-test-slave-nqgh krb5kdc[1617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613588, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:48.707391 1630 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:39:48.708612 1630 webserver.cc:460] Webserver started at http://127.25.71.126:37693/ using document root <none> and password file <none>
I20260928 16:39:48.709336 1630 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:48.709414 1630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:48.709674 1630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:48.712395 1630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "57879701f0c146c39d3438f65f0b66e4"
format_stamp: "Formatted at 2026-09-28 16:39:48 on dist-test-slave-nqgh"
server_key: "7169a280a62cca1bc8953fe890612f88"
server_key_iv: "376b0d9d93dffdebb0dbfc80a0d7bfa0"
server_key_version: "encryptionkey@0"
I20260928 16:39:48.713097 1630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "57879701f0c146c39d3438f65f0b66e4"
format_stamp: "Formatted at 2026-09-28 16:39:48 on dist-test-slave-nqgh"
server_key: "7169a280a62cca1bc8953fe890612f88"
server_key_iv: "376b0d9d93dffdebb0dbfc80a0d7bfa0"
server_key_version: "encryptionkey@0"
I20260928 16:39:48.718032 1630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 16:39:48.721763 1646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:48.723928 1630 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:48.724150 1630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "57879701f0c146c39d3438f65f0b66e4"
format_stamp: "Formatted at 2026-09-28 16:39:48 on dist-test-slave-nqgh"
server_key: "7169a280a62cca1bc8953fe890612f88"
server_key_iv: "376b0d9d93dffdebb0dbfc80a0d7bfa0"
server_key_version: "encryptionkey@0"
I20260928 16:39:48.724315 1630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:48.744557 1630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:48.749054 1630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:48.749310 1630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:48.758241 1630 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:44945
I20260928 16:39:48.758301 1699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:44945 every 8 connection(s)
I20260928 16:39:48.759559 1630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:39:48.760866 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 1630
I20260928 16:39:48.761010 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:39:48.761343 25885 external_mini_cluster.cc:1428] Setting key 5b4388aa8c06e031e2bf15c2ba4b05a2
I20260928 16:39:48.763469 1700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:48.770556 1700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4: Bootstrap starting.
Sep 28 16:39:48 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:48.773135 1700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:48.773953 1700 log.cc:824] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:48.776750 1700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4: No bootstrap required, opened a new log
I20260928 16:39:48.779982 1703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:48.763263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35276 (local address 127.25.71.126:44945)
0928 16:39:48.763813 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:48.763835 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:48.763885 (+ 50us) server_negotiation.cc:407] Connection header received
0928 16:39:48.764778 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:48.764801 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:48.765240 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:48.765688 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:48.766759 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:48.768287 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:48.769514 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:48.770219 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:48.770416 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:48.773711 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:48.773719 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:48.773739 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:48.773778 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:48.775923 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:48.776595 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:48.776601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:48.776608 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:48.776768 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:48.777162 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:48.777165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:48.777167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:48.777750 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:48.777969 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:48.778420 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 16:39:48.778832 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":160,"threads_started":1}
I20260928 16:39:48.781615 1700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879701f0c146c39d3438f65f0b66e4" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44945 } }
I20260928 16:39:48.781922 1700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:39:48.781962 1700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57879701f0c146c39d3438f65f0b66e4, State: Initialized, Role: FOLLOWER
I20260928 16:39:48.782475 1700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879701f0c146c39d3438f65f0b66e4" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44945 } }
I20260928 16:39:48.782617 1700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:39:48.782701 1700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:39:48.782805 1700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:48.784004 1700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879701f0c146c39d3438f65f0b66e4" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44945 } }
I20260928 16:39:48.784374 1700 leader_election.cc:302] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57879701f0c146c39d3438f65f0b66e4; no voters:
I20260928 16:39:48.784703 1700 leader_election.cc:288] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:39:48.784976 1705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:48.785378 1705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [term 1 LEADER]: Becoming Leader. State: Replica: 57879701f0c146c39d3438f65f0b66e4, State: Running, Role: LEADER
I20260928 16:39:48.785817 1705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879701f0c146c39d3438f65f0b66e4" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44945 } }
I20260928 16:39:48.786090 1700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:48.788105 1707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57879701f0c146c39d3438f65f0b66e4. Latest consensus state: current_term: 1 leader_uuid: "57879701f0c146c39d3438f65f0b66e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879701f0c146c39d3438f65f0b66e4" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44945 } } }
I20260928 16:39:48.788287 1707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:48.788573 1706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57879701f0c146c39d3438f65f0b66e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879701f0c146c39d3438f65f0b66e4" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44945 } } }
I20260928 16:39:48.789036 1706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:48.788753 1720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:39:48.793411 1720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:39:48.801971 1720 catalog_manager.cc:1347] Generated new cluster ID: 46e433f90e26468790c4c9ba706a77af
I20260928 16:39:48.802084 1720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:39:48.831962 1720 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:39:48.833470 1720 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:39:48.844961 1720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4: Generated new TSK 0
I20260928 16:39:48.846076 1720 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
I20260928 16:39:48.890822 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:44945
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:49.027504 1726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:49.027912 1726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:49.027984 1726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:49.032970 1726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:49.033100 1726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:49.033178 1726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:39:49.040063 1726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:49.041684 1726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:49.043130 1726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:49.055701 1734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:49.055769 1726 server_base.cc:1034] running on GCE node
W20260928 16:39:49.055701 1732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:49.055749 1731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:49.056653 1726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:49.057579 1726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:49.058804 1726 hybrid_clock.cc:648] HybridClock initialized: now 1790613589058751 us; error 77 us; skew 500 ppm
Sep 28 16:39:49 dist-test-slave-nqgh krb5kdc[1617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613589, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:49.062853 1726 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:39:49.064227 1726 webserver.cc:460] Webserver started at http://127.25.71.65:40923/ using document root <none> and password file <none>
I20260928 16:39:49.064981 1726 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:49.065068 1726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:49.065325 1726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:49.068137 1726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "c3c8ea5463be4418b1657e28bf027a0f"
format_stamp: "Formatted at 2026-09-28 16:39:49 on dist-test-slave-nqgh"
server_key: "29acf7b7d04f76c41b084b8bd4ff9cdd"
server_key_iv: "cc1f6db80da9afd8861e8f971a5c6f18"
server_key_version: "encryptionkey@0"
I20260928 16:39:49.068881 1726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "c3c8ea5463be4418b1657e28bf027a0f"
format_stamp: "Formatted at 2026-09-28 16:39:49 on dist-test-slave-nqgh"
server_key: "29acf7b7d04f76c41b084b8bd4ff9cdd"
server_key_iv: "cc1f6db80da9afd8861e8f971a5c6f18"
server_key_version: "encryptionkey@0"
I20260928 16:39:49.074023 1726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:39:49.078270 1741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:49.080925 1726 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260928 16:39:49.081173 1726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "c3c8ea5463be4418b1657e28bf027a0f"
format_stamp: "Formatted at 2026-09-28 16:39:49 on dist-test-slave-nqgh"
server_key: "29acf7b7d04f76c41b084b8bd4ff9cdd"
server_key_iv: "cc1f6db80da9afd8861e8f971a5c6f18"
server_key_version: "encryptionkey@0"
I20260928 16:39:49.081354 1726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:49.110006 1726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:49.114271 1726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:49.114511 1726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:49.115322 1726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:49.116462 1726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:49.116531 1726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:49.116600 1726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:49.116623 1726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:49.127241 1726 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:41233
I20260928 16:39:49.127466 1856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:41233 every 8 connection(s)
I20260928 16:39:49.128443 1726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:39:49.130180 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 1726
I20260928 16:39:49.130314 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:39:49.130707 25885 external_mini_cluster.cc:1428] Setting key 0386dd9dfa655cee312261a1fed5b6f7
WARNING: no policy specified for kudu/127.25.71.66@KRBTEST.COM; defaulting to no policy
Sep 28 16:39:49 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613589, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.66@KRBTEST.COM" created.
I20260928 16:39:49.148831 1703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:49.131370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:33639 (local address 127.25.71.126:44945)
0928 16:39:49.133020 (+ 1650us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:49.133026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:49.133048 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:39:49.134597 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:49.134601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:49.134709 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:49.134820 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:49.136886 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:49.137549 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:49.139398 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:49.139714 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:49.139840 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:49.143564 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:49.143569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:49.143572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:49.143617 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:49.145495 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:49.146368 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:49.146372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:49.146374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:49.146476 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:49.146968 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:49.146972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:49.146973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:49.147304 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:49.147421 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:49.148403 (+ 982us) server_negotiation.cc:299] Negotiation successful
0928 16:39:49.148631 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1513}
I20260928 16:39:49.150158 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:49.131795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.65:33639)
0928 16:39:49.132588 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:49.132669 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:49.134217 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:49.135151 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:49.135165 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:49.135876 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:49.136659 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:49.136680 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:49.137833 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:49.137837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:49.139194 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:49.139210 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:49.139923 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:49.139928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:49.140179 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:49.140937 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:49.140961 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:49.143196 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:49.145749 (+ 2553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:49.145759 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:49.145785 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:49.146230 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:49.146597 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:49.146601 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:49.146603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:49.146863 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:49.147538 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:49.147550 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:49.148029 (+ 479us) client_negotiation.cc:769] Sending connection context
0928 16:39:49.148299 (+ 270us) client_negotiation.cc:240] Negotiation successful
0928 16:39:49.148714 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":183,"threads_started":1}
I20260928 16:39:49.152107 1857 heartbeater.cc:347] Connected to a master server at 127.25.71.126:44945
I20260928 16:39:49.152522 1857 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:49.153342 1857 heartbeater.cc:511] Master 127.25.71.126:44945 requested a full tablet report, sending...
I20260928 16:39:49.155697 1660 ts_manager.cc:194] Registered new tserver with Master: c3c8ea5463be4418b1657e28bf027a0f (127.25.71.65:41233)
I20260928 16:39:49.157321 1660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:33639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
I20260928 16:39:49.167156 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:44945
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:49.299644 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:49.299988 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:49.300036 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:49.304524 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:49.304610 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:49.304677 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:39:49.310552 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:49.311780 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:49.312881 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:49.323973 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:49.323920 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:49.324143 1862 server_base.cc:1034] running on GCE node
W20260928 16:39:49.323899 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:49.324756 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:49.325491 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:49.326716 1862 hybrid_clock.cc:648] HybridClock initialized: now 1790613589326697 us; error 40 us; skew 500 ppm
Sep 28 16:39:49 dist-test-slave-nqgh krb5kdc[1617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613589, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:49.330065 1862 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:39:49.331229 1862 webserver.cc:460] Webserver started at http://127.25.71.66:43687/ using document root <none> and password file <none>
I20260928 16:39:49.331884 1862 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:49.331960 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:49.332173 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:49.334357 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "1cdc6f2055174d8b9243c8f5fc4b6b55"
format_stamp: "Formatted at 2026-09-28 16:39:49 on dist-test-slave-nqgh"
server_key: "bdbb4a7c8aba15149d3124f2e1589f59"
server_key_iv: "cab4060ab7928cbdc505d2304bf2baca"
server_key_version: "encryptionkey@0"
I20260928 16:39:49.334980 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "1cdc6f2055174d8b9243c8f5fc4b6b55"
format_stamp: "Formatted at 2026-09-28 16:39:49 on dist-test-slave-nqgh"
server_key: "bdbb4a7c8aba15149d3124f2e1589f59"
server_key_iv: "cab4060ab7928cbdc505d2304bf2baca"
server_key_version: "encryptionkey@0"
I20260928 16:39:49.339392 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:49.342682 1878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:49.344568 1862 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:49.344739 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "1cdc6f2055174d8b9243c8f5fc4b6b55"
format_stamp: "Formatted at 2026-09-28 16:39:49 on dist-test-slave-nqgh"
server_key: "bdbb4a7c8aba15149d3124f2e1589f59"
server_key_iv: "cab4060ab7928cbdc505d2304bf2baca"
server_key_version: "encryptionkey@0"
I20260928 16:39:49.344866 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:49.368433 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:49.372313 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:49.372536 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:49.373180 1862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:49.374116 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:49.374169 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:49.374233 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:49.374260 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:49.385458 1862 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:42773
I20260928 16:39:49.385649 1994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:42773 every 8 connection(s)
I20260928 16:39:49.386561 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:39:49.396281 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 1862
I20260928 16:39:49.396409 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:39:49.396690 25885 external_mini_cluster.cc:1428] Setting key 97916056a0903f3eb71b0ed8cb72b573
Sep 28 16:39:49 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613589, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:49.404825 1703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:49.389376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:51037 (local address 127.25.71.126:44945)
0928 16:39:49.389611 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:49.389616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:49.390613 (+ 997us) server_negotiation.cc:407] Connection header received
0928 16:39:49.392392 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:49.392396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:49.392475 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:49.392559 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:49.394192 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:49.394841 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:49.396246 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:49.396541 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:49.396644 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:49.400048 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:49.400053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:49.400057 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:49.400107 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:49.401700 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:49.402499 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:49.402504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:49.402506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:49.402666 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:49.403234 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:49.403237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:49.403239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:49.403538 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:49.403659 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:49.404441 (+ 782us) server_negotiation.cc:299] Negotiation successful
0928 16:39:49.404636 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 16:39:49.405670 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:49.389715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.66:51037)
0928 16:39:49.390382 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:49.390449 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:49.392075 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:49.392828 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:49.392837 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:49.393326 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:49.393994 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:49.394011 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:49.394982 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:49.394986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:49.396086 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:49.396102 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:49.396728 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:49.396733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:49.396974 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:49.397706 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:49.397725 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:49.399773 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:49.401889 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:49.401898 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:49.401922 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:49.402339 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:49.402833 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:49.402836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:49.402894 (+ 58us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:49.403073 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:49.403680 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:49.403687 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:49.404080 (+ 393us) client_negotiation.cc:769] Sending connection context
0928 16:39:49.404286 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 16:39:49.404671 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":157,"threads_started":1}
I20260928 16:39:49.407131 1995 heartbeater.cc:347] Connected to a master server at 127.25.71.126:44945
I20260928 16:39:49.407604 1995 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:49.408519 1995 heartbeater.cc:511] Master 127.25.71.126:44945 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.25.71.67@KRBTEST.COM; defaulting to no policy
I20260928 16:39:49.410164 1660 ts_manager.cc:194] Registered new tserver with Master: 1cdc6f2055174d8b9243c8f5fc4b6b55 (127.25.71.66:42773)
I20260928 16:39:49.410933 1660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:51037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
I20260928 16:39:49.431095 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:44945
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:49.565603 2000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:49.566013 2000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:49.566072 2000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:49.570724 2000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:49.570833 2000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:49.570904 2000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:39:49.577236 2000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:49.578837 2000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:49.580200 2000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:49.592618 2005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:49.592640 2008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:49.592653 2006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:49.592995 2000 server_base.cc:1034] running on GCE node
I20260928 16:39:49.593663 2000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:49.594579 2000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:49.595826 2000 hybrid_clock.cc:648] HybridClock initialized: now 1790613589595793 us; error 48 us; skew 500 ppm
Sep 28 16:39:49 dist-test-slave-nqgh krb5kdc[1617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613589, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:49.599700 2000 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:39:49.601048 2000 webserver.cc:460] Webserver started at http://127.25.71.67:41563/ using document root <none> and password file <none>
I20260928 16:39:49.601810 2000 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:49.601888 2000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:49.602126 2000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:49.604961 2000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "335e8663271a4a2c9a5c0932ea2403dc"
format_stamp: "Formatted at 2026-09-28 16:39:49 on dist-test-slave-nqgh"
server_key: "f39a29f28fc15d8ecda4d8f58251877f"
server_key_iv: "cf199cee4dff63f337c40e9cb7ab5e68"
server_key_version: "encryptionkey@0"
I20260928 16:39:49.605664 2000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "335e8663271a4a2c9a5c0932ea2403dc"
format_stamp: "Formatted at 2026-09-28 16:39:49 on dist-test-slave-nqgh"
server_key: "f39a29f28fc15d8ecda4d8f58251877f"
server_key_iv: "cf199cee4dff63f337c40e9cb7ab5e68"
server_key_version: "encryptionkey@0"
I20260928 16:39:49.611490 2000 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260928 16:39:49.615446 2015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:49.617448 2000 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:49.617661 2000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "335e8663271a4a2c9a5c0932ea2403dc"
format_stamp: "Formatted at 2026-09-28 16:39:49 on dist-test-slave-nqgh"
server_key: "f39a29f28fc15d8ecda4d8f58251877f"
server_key_iv: "cf199cee4dff63f337c40e9cb7ab5e68"
server_key_version: "encryptionkey@0"
I20260928 16:39:49.617825 2000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:49.639052 2000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:49.643530 2000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:49.643767 2000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:49.644600 2000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:49.645779 2000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:49.645836 2000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:49.645888 2000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:49.645908 2000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:49.657866 2000 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:36369
I20260928 16:39:49.657933 2130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:36369 every 8 connection(s)
I20260928 16:39:49.659157 2000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:39:49.660023 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 2000
I20260928 16:39:49.660181 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:39:49.660529 25885 external_mini_cluster.cc:1428] Setting key d9b003d8a5eb77a4e78ef2dfa87bad55
Sep 28 16:39:49 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613589, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:39:49.677593 1703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:49.661646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:46701 (local address 127.25.71.126:44945)
0928 16:39:49.661851 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:49.661927 (+ 76us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:49.663341 (+ 1414us) server_negotiation.cc:407] Connection header received
0928 16:39:49.664995 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:49.664998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:49.665060 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:49.665146 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:49.666818 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:49.667406 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:49.668885 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:49.669150 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:49.669266 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:49.672651 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:49.672655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:49.672657 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:49.672696 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:49.674442 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:49.675345 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:49.675350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:49.675351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:49.675476 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:49.676071 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:49.676075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:49.676076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:49.676335 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:49.676451 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:49.677297 (+ 846us) server_negotiation.cc:299] Negotiation successful
0928 16:39:49.677459 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:39:49.678887 2133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:49.662358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.67:46701)
0928 16:39:49.663127 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:49.663197 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:49.664721 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:49.665390 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:49.665399 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:49.665895 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:49.666556 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:49.666572 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:49.667566 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:49.667570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:49.668691 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:49.668705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:49.669347 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:49.669350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:49.669535 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:49.670317 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:49.670336 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:49.672357 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:49.674667 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:49.674676 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:49.674701 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:49.675192 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:49.675622 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:49.675625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:49.675628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:49.675910 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:49.676485 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:49.676495 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:49.676979 (+ 484us) client_negotiation.cc:769] Sending connection context
0928 16:39:49.677211 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 16:39:49.677615 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":287,"threads_started":1}
I20260928 16:39:49.681350 2131 heartbeater.cc:347] Connected to a master server at 127.25.71.126:44945
I20260928 16:39:49.681800 2131 heartbeater.cc:464] Registering TS with master...
I20260928 16:39:49.682435 2131 heartbeater.cc:511] Master 127.25.71.126:44945 requested a full tablet report, sending...
I20260928 16:39:49.683908 1660 ts_manager.cc:194] Registered new tserver with Master: 335e8663271a4a2c9a5c0932ea2403dc (127.25.71.67:36369)
I20260928 16:39:49.684535 1660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:46701
I20260928 16:39:49.689836 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.25.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.65.keytab.
Entry for principal oryx/127.25.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.65.keytab.
WARNING: no policy specified for oryx/127.25.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.71.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.66.keytab.
Entry for principal oryx/127.25.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.66.keytab.
WARNING: no policy specified for oryx/127.25.71.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.71.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.67.keytab.
Entry for principal oryx/127.25.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.67.keytab.
I20260928 16:39:49.790892 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 1726
I20260928 16:39:49.800552 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 1862
I20260928 16:39:49.809477 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 2000
I20260928 16:39:49.816042 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 1630
I20260928 16:39:49.823562 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:44945
--webserver_interface=127.25.71.126
--webserver_port=37693
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:44945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:39:49.965618 2140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:49.966025 2140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:49.966077 2140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:49.970870 2140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:49.970989 2140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:49.971009 2140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:49.971026 2140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:49.971043 2140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:39:49.978097 2140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:44945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:44945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=37693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:49.979705 2140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:49.981056 2140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:49.991623 2145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:49.991582 2148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:49.991611 2146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:49.991699 2140 server_base.cc:1034] running on GCE node
I20260928 16:39:49.992722 2140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:49.993978 2140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:49.995189 2140 hybrid_clock.cc:648] HybridClock initialized: now 1790613589995161 us; error 51 us; skew 500 ppm
Sep 28 16:39:49 dist-test-slave-nqgh krb5kdc[1617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613589, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:49.998924 2140 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:39:50.000427 2140 webserver.cc:460] Webserver started at http://127.25.71.126:37693/ using document root <none> and password file <none>
I20260928 16:39:50.001186 2140 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:50.001279 2140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:50.006330 2140 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:39:50.009136 2155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:50.010951 2140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 16:39:50.011111 2140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "57879701f0c146c39d3438f65f0b66e4"
format_stamp: "Formatted at 2026-09-28 16:39:48 on dist-test-slave-nqgh"
server_key: "7169a280a62cca1bc8953fe890612f88"
server_key_iv: "376b0d9d93dffdebb0dbfc80a0d7bfa0"
server_key_version: "encryptionkey@0"
I20260928 16:39:50.011726 2140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:50.026605 2140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:50.030241 2140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:50.030443 2140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:50.038489 2140 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:44945
I20260928 16:39:50.038565 2208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:44945 every 8 connection(s)
I20260928 16:39:50.039811 2140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:39:50.043217 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 2140
I20260928 16:39:50.043673 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:41233
--local_ip_for_outbound_sockets=127.25.71.65
--tserver_master_addrs=127.25.71.126:44945
--webserver_port=40923
--webserver_interface=127.25.71.65
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.25.71.65
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:39:50.046154 2209 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:39:50.048564 2209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4: Bootstrap starting.
I20260928 16:39:50.069686 2209 log.cc:824] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:50.078775 2209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:39:50.079286 2209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4: Bootstrap complete.
I20260928 16:39:50.084444 2209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879701f0c146c39d3438f65f0b66e4" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44945 } }
I20260928 16:39:50.085026 2209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57879701f0c146c39d3438f65f0b66e4, State: Initialized, Role: FOLLOWER
I20260928 16:39:50.085800 2209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879701f0c146c39d3438f65f0b66e4" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44945 } }
I20260928 16:39:50.085994 2209 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:39:50.086092 2209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:39:50.086231 2209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:39:50.089252 2209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879701f0c146c39d3438f65f0b66e4" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44945 } }
I20260928 16:39:50.089771 2209 leader_election.cc:302] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57879701f0c146c39d3438f65f0b66e4; no voters:
I20260928 16:39:50.090209 2209 leader_election.cc:288] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 16:39:50.090345 2213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:39:50.090590 2213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [term 2 LEADER]: Becoming Leader. State: Replica: 57879701f0c146c39d3438f65f0b66e4, State: Running, Role: LEADER
I20260928 16:39:50.091017 2213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879701f0c146c39d3438f65f0b66e4" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44945 } }
I20260928 16:39:50.091864 2209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:50.092660 2214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "57879701f0c146c39d3438f65f0b66e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879701f0c146c39d3438f65f0b66e4" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44945 } } }
I20260928 16:39:50.092828 2214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:50.093178 2215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57879701f0c146c39d3438f65f0b66e4. Latest consensus state: current_term: 2 leader_uuid: "57879701f0c146c39d3438f65f0b66e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879701f0c146c39d3438f65f0b66e4" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44945 } } }
I20260928 16:39:50.093276 2215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:50.097921 2229 catalog_manager.cc:1259] Loaded cluster ID: 46e433f90e26468790c4c9ba706a77af
I20260928 16:39:50.098035 2229 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4: loading cluster ID for follower catalog manager: success
I20260928 16:39:50.100687 2229 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4: acquiring CA information for follower catalog manager: success
I20260928 16:39:50.102586 2229 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:39:50.102885 2220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:39:50.104208 2220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:39:50.104700 2220 catalog_manager.cc:1259] Loaded cluster ID: 46e433f90e26468790c4c9ba706a77af
I20260928 16:39:50.104771 2220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:39:50.105533 2220 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:39:50.106010 2220 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 57879701f0c146c39d3438f65f0b66e4: Loaded TSK: 0
I20260928 16:39:50.107136 2220 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:39:50.187129 2211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:50.187610 2211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:50.187683 2211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:50.192606 2211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:50.192725 2211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:50.192808 2211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:39:50.199705 2211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.65.keytab
--principal=oryx/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:41233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=40923
--tserver_master_addrs=127.25.71.126:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:50.201350 2211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:50.202829 2211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:50.215484 2236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:50.215471 2237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:50.215471 2239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:50.216952 2211 server_base.cc:1034] running on GCE node
I20260928 16:39:50.217777 2211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:50.218861 2211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:50.220168 2211 hybrid_clock.cc:648] HybridClock initialized: now 1790613590220140 us; error 57 us; skew 500 ppm
Sep 28 16:39:50 dist-test-slave-nqgh krb5kdc[1617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613590, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:50.224061 2211 init.cc:377] Logged in from keytab as oryx/127.25.71.65@KRBTEST.COM (short username oryx)
I20260928 16:39:50.225368 2211 webserver.cc:460] Webserver started at http://127.25.71.65:40923/ using document root <none> and password file <none>
I20260928 16:39:50.226125 2211 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:50.226234 2211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:50.232033 2211 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:50.235329 2246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:50.237206 2211 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:50.237417 2211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "c3c8ea5463be4418b1657e28bf027a0f"
format_stamp: "Formatted at 2026-09-28 16:39:49 on dist-test-slave-nqgh"
server_key: "29acf7b7d04f76c41b084b8bd4ff9cdd"
server_key_iv: "cc1f6db80da9afd8861e8f971a5c6f18"
server_key_version: "encryptionkey@0"
I20260928 16:39:50.238198 2211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:50.274948 2211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:50.279487 2211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:50.279747 2211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:50.280568 2211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:50.281735 2211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:50.281787 2211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:50.281850 2211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:50.281879 2211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:50.293404 2211 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:41233
I20260928 16:39:50.293422 2360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:41233 every 8 connection(s)
I20260928 16:39:50.294849 2211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:39:50.303115 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 2211
I20260928 16:39:50.303579 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:42773
--local_ip_for_outbound_sockets=127.25.71.66
--tserver_master_addrs=127.25.71.126:44945
--webserver_port=43687
--webserver_interface=127.25.71.66
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.25.71.66
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:39:50 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:50 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:50.312449 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:50.297691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.65:38201)
0928 16:39:50.298304 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:50.298366 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:50.299846 (+ 1480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:50.301674 (+ 1828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:50.301683 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:50.302144 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:50.302784 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:50.302805 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.304780 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.304785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:50.305852 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:50.305869 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.306884 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.306890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:50.307130 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:50.307916 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:50.307939 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:50.311046 (+ 3107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":178,"threads_started":1}
W20260928 16:39:50.312436 2364 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:50.297675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:38201 (local address 127.25.71.126:44945)
0928 16:39:50.298296 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:50.298317 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:50.298564 (+ 247us) server_negotiation.cc:407] Connection header received
0928 16:39:50.300465 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:50.300484 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:50.301002 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:50.301455 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:50.303176 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.304540 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.306040 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.306618 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.306891 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:50.311379 (+ 4488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.65:38201: BlockingRecv error: recv got EOF from 127.25.71.65:38201 (error 108)
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":170,"threads_started":1}
W20260928 16:39:50.313539 2361 heartbeater.cc:644] Failed to heartbeat to 127.25.71.126:44945 (0 consecutive failures): Not authorized: Failed to ping master at 127.25.71.126:44945: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Sep 28 16:39:50 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:50 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:50.321596 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:50.314234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.65:44705)
0928 16:39:50.314491 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:50.314510 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:50.314694 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:50.315035 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:50.315038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:50.315261 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:50.315425 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:50.315433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.316422 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.316426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:50.317424 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:50.317437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.318055 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.318058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:50.318166 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:50.319000 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:50.319015 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:50.321407 (+ 2392us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99}
W20260928 16:39:50.321758 2364 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:50.314394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:44705 (local address 127.25.71.126:44945)
0928 16:39:50.314536 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:50.314540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:50.314607 (+ 67us) server_negotiation.cc:407] Connection header received
0928 16:39:50.314828 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:50.314832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:50.314893 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:50.314978 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:50.315566 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.316271 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.317583 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.317880 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.318032 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:50.321633 (+ 3601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.65:44705: BlockingRecv error: recv got EOF from 127.25.71.65:44705 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 16:39:50 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:50 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:50.329434 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:50.322579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.65:35223)
0928 16:39:50.322799 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:50.322816 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:50.322927 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:50.323310 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:50.323313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:50.323525 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:50.323654 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:50.323661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.324644 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.324648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:50.325476 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:50.325488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.326036 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.326039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:50.326129 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:50.326826 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:50.326840 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:50.329248 (+ 2408us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92}
W20260928 16:39:50.329595 2364 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:50.322726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:35223 (local address 127.25.71.126:44945)
0928 16:39:50.322880 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:50.322884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:50.322900 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:39:50.323113 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:50.323115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:50.323176 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:50.323256 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:50.323821 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.324515 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.325639 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.325887 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.325987 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:50.329468 (+ 3481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.65:35223: BlockingRecv error: recv got EOF from 127.25.71.65:35223 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 16:39:50.329964 2361 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:39:50.435017 2365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:50.435346 2365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:50.435410 2365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:50.439858 2365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:50.439944 2365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:50.440032 2365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:39:50.445919 2365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.66.keytab
--principal=oryx/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:42773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=43687
--tserver_master_addrs=127.25.71.126:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:50.447474 2365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:50.448688 2365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:50.460388 2371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:50.460379 2374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:50.460618 2365 server_base.cc:1034] running on GCE node
W20260928 16:39:50.460399 2370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:50.461591 2365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:50.462505 2365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:50.463730 2365 hybrid_clock.cc:648] HybridClock initialized: now 1790613590463690 us; error 62 us; skew 500 ppm
Sep 28 16:39:50 dist-test-slave-nqgh krb5kdc[1617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613590, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:50.467551 2365 init.cc:377] Logged in from keytab as oryx/127.25.71.66@KRBTEST.COM (short username oryx)
I20260928 16:39:50.468963 2365 webserver.cc:460] Webserver started at http://127.25.71.66:43687/ using document root <none> and password file <none>
I20260928 16:39:50.469761 2365 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:50.469878 2365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:50.475565 2365 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:50.479063 2381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:50.481115 2365 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:39:50.481328 2365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "1cdc6f2055174d8b9243c8f5fc4b6b55"
format_stamp: "Formatted at 2026-09-28 16:39:49 on dist-test-slave-nqgh"
server_key: "bdbb4a7c8aba15149d3124f2e1589f59"
server_key_iv: "cab4060ab7928cbdc505d2304bf2baca"
server_key_version: "encryptionkey@0"
I20260928 16:39:50.482100 2365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:50.497099 2365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:50.501904 2365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:50.502146 2365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:50.503010 2365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:50.504094 2365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:50.504159 2365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:50.504213 2365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:50.504238 2365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:50.515697 2365 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:42773
I20260928 16:39:50.515811 2494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:42773 every 8 connection(s)
I20260928 16:39:50.517151 2365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:39:50.521567 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 2365
I20260928 16:39:50.522126 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:36369
--local_ip_for_outbound_sockets=127.25.71.67
--tserver_master_addrs=127.25.71.126:44945
--webserver_port=41563
--webserver_interface=127.25.71.67
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.25.71.67
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:39:50 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:50 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:39:50.533751 2364 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:50.519616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:33693 (local address 127.25.71.126:44945)
0928 16:39:50.519784 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:50.519788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:50.520845 (+ 1057us) server_negotiation.cc:407] Connection header received
0928 16:39:50.522713 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:50.522717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:50.522815 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:50.522900 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:50.525087 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.525717 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.527590 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.528010 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.528192 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:50.533600 (+ 5408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.66:33693: BlockingRecv error: recv got EOF from 127.25.71.66:33693 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:39:50.534546 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:50.519902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.66:33693)
0928 16:39:50.520576 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:50.520650 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:50.522362 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:50.523171 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:50.523181 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:50.523823 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:50.524767 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:50.524795 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.525949 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.525952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:50.527338 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:50.527358 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.528325 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.528337 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:50.528593 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:50.529447 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:50.529471 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:50.533308 (+ 3837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":163,"threads_started":1}
W20260928 16:39:50.535380 2495 heartbeater.cc:644] Failed to heartbeat to 127.25.71.126:44945 (0 consecutive failures): Not authorized: Failed to ping master at 127.25.71.126:44945: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Sep 28 16:39:50 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:50 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:50.543169 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:50.536091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.66:60219)
0928 16:39:50.536363 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:50.536381 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:50.536510 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:50.536834 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:50.536836 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:50.537027 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:50.537240 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:50.537249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.538373 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.538377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:50.539202 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:50.539217 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.539867 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.539870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:50.539972 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:50.540703 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:50.540731 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:50.542994 (+ 2263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153}
W20260928 16:39:50.543571 2364 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:50.536244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:60219 (local address 127.25.71.126:44945)
0928 16:39:50.536456 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:50.536462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:50.536486 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:39:50.536612 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:50.536616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:50.536679 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:50.536789 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:50.537365 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.538216 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.539343 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.539714 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.539823 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:50.543238 (+ 3415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.66:60219: BlockingRecv error: recv got EOF from 127.25.71.66:60219 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 28 16:39:50 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:50 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:50.551010 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:50.544083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.66:38849)
0928 16:39:50.544257 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:50.544277 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:50.544395 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:50.544761 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:50.544764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:50.544993 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:50.545136 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:50.545143 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.546134 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.546138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:50.547034 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:50.547046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.547566 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.547569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:50.547653 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:50.548266 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:50.548281 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:50.550819 (+ 2538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 16:39:50.551350 2495 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:39:50.551532 2364 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:50.544179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:38849 (local address 127.25.71.126:44945)
0928 16:39:50.544312 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:50.544316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:50.544331 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:39:50.544531 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:50.544574 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:50.544628 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:50.544709 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:50.545302 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.545987 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.547176 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.547429 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.547533 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:50.551048 (+ 3515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.66:38849: BlockingRecv error: recv got EOF from 127.25.71.66:38849 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 16:39:50.644953 2498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:50.645265 2498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:50.645329 2498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:50.649637 2498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:50.649729 2498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:50.649818 2498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:39:50.655464 2498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/krb5kdc/oryx_127.25.71.67.keytab
--principal=oryx/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:36369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=41563
--tserver_master_addrs=127.25.71.126:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:50.656719 2498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:50.657799 2498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:39:50.669024 2498 server_base.cc:1034] running on GCE node
W20260928 16:39:50.669011 2506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:50.669018 2504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:50.669011 2503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:50.670118 2498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:50.670991 2498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:50.672299 2498 hybrid_clock.cc:648] HybridClock initialized: now 1790613590672270 us; error 55 us; skew 500 ppm
Sep 28 16:39:50 dist-test-slave-nqgh krb5kdc[1617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613590, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:50.676111 2498 init.cc:377] Logged in from keytab as oryx/127.25.71.67@KRBTEST.COM (short username oryx)
I20260928 16:39:50.677255 2498 webserver.cc:460] Webserver started at http://127.25.71.67:41563/ using document root <none> and password file <none>
I20260928 16:39:50.677884 2498 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:50.677979 2498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:50.682883 2498 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:39:50.685954 2513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:50.688052 2498 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:39:50.688259 2498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "335e8663271a4a2c9a5c0932ea2403dc"
format_stamp: "Formatted at 2026-09-28 16:39:49 on dist-test-slave-nqgh"
server_key: "f39a29f28fc15d8ecda4d8f58251877f"
server_key_iv: "cf199cee4dff63f337c40e9cb7ab5e68"
server_key_version: "encryptionkey@0"
I20260928 16:39:50.688941 2498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:50.709372 2498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:50.713586 2498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:50.713807 2498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:50.714573 2498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:39:50.715626 2498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:39:50.715680 2498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:50.715739 2498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:39:50.715768 2498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:50.727224 2498 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:36369
I20260928 16:39:50.727432 2626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:36369 every 8 connection(s)
I20260928 16:39:50.728444 2498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:39:50.732316 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 2498
Sep 28 16:39:50 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:50 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:39:50.743261 2364 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:50.731106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:49041 (local address 127.25.71.126:44945)
0928 16:39:50.731343 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:50.731349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:50.732649 (+ 1300us) server_negotiation.cc:407] Connection header received
0928 16:39:50.734544 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:50.734549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:50.734617 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:50.734707 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:50.736418 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.736974 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.738567 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.738850 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.738937 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:50.743059 (+ 4122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.67:49041: BlockingRecv error: recv got EOF from 127.25.71.67:49041 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 16:39:50.743966 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:50.731644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.67:49041)
0928 16:39:50.732409 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:50.732500 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:50.734233 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:50.735222 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:50.735233 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:50.735672 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:50.736214 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:50.736233 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.737218 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.737221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:50.738409 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:50.738421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.739015 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.739019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:50.739211 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:50.739789 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:50.739820 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:50.742893 (+ 3073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":213,"threads_started":1}
W20260928 16:39:50.744681 2627 heartbeater.cc:644] Failed to heartbeat to 127.25.71.126:44945 (0 consecutive failures): Not authorized: Failed to ping master at 127.25.71.126:44945: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
I20260928 16:39:50.745184 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:50.733787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.25.71.126:44945)
0928 16:39:50.734112 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:50.734117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:50.734131 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:39:50.734184 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:50.734187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:50.734237 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:50.734366 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:50.735471 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.736115 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.737109 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.737394 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.737477 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:50.739206 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:50.739221 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:50.739245 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:50.739276 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:50.741777 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:50.742459 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:50.742468 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:50.742477 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:50.742711 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:50.743066 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:50.743150 (+ 84us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:50.743152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:50.743761 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:50.743973 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:50.744740 (+ 767us) server_negotiation.cc:299] Negotiation successful
0928 16:39:50.745037 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":103,"threads_started":1}
Sep 28 16:39:50 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:50 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:50.751796 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:50.745260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.67:54619)
0928 16:39:50.745471 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:50.745487 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:50.745595 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:50.745838 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:50.745841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:50.746032 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:50.746156 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:50.746162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.747098 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.747102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:50.747843 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:50.747857 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.748377 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.748382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:50.748483 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:50.749117 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:50.749129 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:50.751614 (+ 2485us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137}
W20260928 16:39:50.752408 2364 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:50.745333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:54619 (local address 127.25.71.126:44945)
0928 16:39:50.745469 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:50.745473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:50.745531 (+ 58us) server_negotiation.cc:407] Connection header received
0928 16:39:50.745684 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:50.745687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:50.745736 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:50.745800 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:50.746332 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.746970 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.747970 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.748261 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.748358 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:50.751829 (+ 3471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.67:54619: BlockingRecv error: recv got EOF from 127.25.71.67:54619 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 16:39:50 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:50 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:50.758862 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:50.752713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.67:52963)
0928 16:39:50.752885 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:50.752901 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:50.753008 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:50.753419 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:50.753422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:50.753617 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:50.753719 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:50.753724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.754573 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.754576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:50.755174 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:50.755184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.755635 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.755639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:50.755723 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:50.756275 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:50.756289 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:50.758677 (+ 2388us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78}
W20260928 16:39:50.759174 2364 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:50.752871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:52963 (local address 127.25.71.126:44945)
0928 16:39:50.753197 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:50.753201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:50.753213 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:39:50.753257 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:50.753259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:50.753304 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:50.753376 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:50.753835 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.754414 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.755294 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:50.755522 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:50.755615 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:50.759003 (+ 3388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.67:52963: BlockingRecv error: recv got EOF from 127.25.71.67:52963 (error 108)
Metrics: {"server-negotiator.queue_time_us":233}
W20260928 16:39:50.759181 2627 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 16:39:51 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:51 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:51.338603 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:51.331119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.65:42181)
0928 16:39:51.331490 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:51.331513 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:51.331678 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:51.332141 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:51.332144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:51.332396 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:51.332549 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:51.332566 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:51.333640 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:51.333645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:51.334421 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:51.334437 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:51.335173 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:51.335176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:51.335255 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:51.335958 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:51.335976 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:51.338375 (+ 2399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":88,"threads_started":1}
W20260928 16:39:51.338824 2633 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:51.331325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:42181 (local address 127.25.71.126:44945)
0928 16:39:51.331731 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:51.331735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:51.331756 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:39:51.331813 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:51.331816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:51.331873 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:51.331998 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:51.332766 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:51.333437 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:51.334694 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:51.335015 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:51.335149 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:51.338676 (+ 3527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.65:42181: BlockingRecv error: recv got EOF from 127.25.71.65:42181 (error 108)
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":111,"threads_started":1}
Sep 28 16:39:51 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:51 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:51.559434 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:51.552142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.66:34627)
0928 16:39:51.552600 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:51.552622 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:51.552822 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:51.553131 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:51.553134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:51.553378 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:51.553519 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:51.553526 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:51.554501 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:51.554504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:51.555193 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:51.555205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:51.555844 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:51.555850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:51.556018 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:51.556702 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:51.556718 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:51.559252 (+ 2534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":96,"threads_started":1}
W20260928 16:39:51.559643 2633 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:51.552408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:34627 (local address 127.25.71.126:44945)
0928 16:39:51.552626 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:51.552631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:51.552726 (+ 95us) server_negotiation.cc:407] Connection header received
0928 16:39:51.552941 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:51.552943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:51.552996 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:51.553079 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:51.553693 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:51.554376 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:51.555325 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:51.555600 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:51.555725 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:51.559473 (+ 3748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.66:34627: BlockingRecv error: recv got EOF from 127.25.71.66:34627 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 28 16:39:51 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:51 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:51.766991 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:51.760107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.67:47625)
0928 16:39:51.760452 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:51.760468 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:51.760599 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:51.760948 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:51.760950 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:51.761141 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:51.761265 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:51.761271 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:51.762238 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:51.762243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:51.763001 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:51.763014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:51.763491 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:51.763493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:51.763559 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:51.764182 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:51.764194 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:51.766785 (+ 2591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":95,"threads_started":1}
W20260928 16:39:51.767270 2633 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:51.760234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:47625 (local address 127.25.71.126:44945)
0928 16:39:51.760473 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:51.760478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:51.760591 (+ 113us) server_negotiation.cc:407] Connection header received
0928 16:39:51.760706 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:51.760709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:51.760773 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:51.760857 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:51.761406 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:51.762098 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:51.763135 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:51.763387 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:51.763472 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:51.767083 (+ 3611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.67:47625: BlockingRecv error: recv got EOF from 127.25.71.67:47625 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 28 16:39:52 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:52 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:52.347277 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:52.339947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.65:46917)
0928 16:39:52.340370 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:52.340398 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:52.340551 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:52.341012 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:52.341015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:52.341214 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:52.341339 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:52.341359 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:52.342327 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:52.342330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:52.343062 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:52.343075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:52.343681 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:52.343685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:52.343790 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:52.344472 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:52.344494 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:52.346983 (+ 2489us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":121,"threads_started":1}
W20260928 16:39:52.347404 2637 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:52.340116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:46917 (local address 127.25.71.126:44945)
0928 16:39:52.340438 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:52.340442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:52.340485 (+ 43us) server_negotiation.cc:407] Connection header received
0928 16:39:52.340657 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:52.340660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:52.340735 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:52.340845 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:52.341525 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:52.342170 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:52.343215 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:52.343523 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:52.343624 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:52.347229 (+ 3605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.65:46917: BlockingRecv error: recv got EOF from 127.25.71.65:46917 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":117,"threads_started":1}
Sep 28 16:39:52 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:52 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:52.567548 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:52.560602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.66:44885)
0928 16:39:52.561057 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:52.561075 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:52.561237 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:52.561757 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:52.561761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:52.562085 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:52.562256 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:52.562263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:52.563343 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:52.563346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:52.563995 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:52.564005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:52.564572 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:52.564575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:52.564657 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:52.565348 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:52.565361 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:52.567370 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":122,"threads_started":1}
W20260928 16:39:52.568061 2637 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:52.560840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:44885 (local address 127.25.71.126:44945)
0928 16:39:52.561036 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:52.561043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:52.561192 (+ 149us) server_negotiation.cc:407] Connection header received
0928 16:39:52.561427 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:52.561431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:52.561514 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:52.561679 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:52.562401 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:52.563203 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:52.564146 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:52.564417 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:52.564516 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:52.567923 (+ 3407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.66:44885: BlockingRecv error: recv got EOF from 127.25.71.66:44885 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 16:39:52 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:52 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:52.776219 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:52.769677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.67:41801)
0928 16:39:52.770041 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:52.770058 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:52.770245 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:52.770558 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:52.770561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:52.770823 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:52.770957 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:52.770964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:52.771887 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:52.771890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:52.772515 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:52.772523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:52.773021 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:52.773024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:52.773111 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:52.773790 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:52.773803 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:52.776000 (+ 2197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":93,"threads_started":1}
W20260928 16:39:52.776436 2637 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:52.769881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:41801 (local address 127.25.71.126:44945)
0928 16:39:52.770051 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:52.770055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:52.770186 (+ 131us) server_negotiation.cc:407] Connection header received
0928 16:39:52.770368 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:52.770370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:52.770427 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:52.770505 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:52.771088 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:52.771735 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:52.772638 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:52.772885 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:52.772972 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:52.776244 (+ 3272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.67:41801: BlockingRecv error: recv got EOF from 127.25.71.67:41801 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 16:39:53 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:53 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:53.355526 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:53.348478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.65:49023)
0928 16:39:53.348928 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:53.348961 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:53.349153 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:53.349512 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:53.349515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:53.349811 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:53.349959 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:53.349969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:53.350978 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:53.350982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:53.351663 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:53.351674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:53.352251 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:53.352253 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:53.352327 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:53.352896 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:53.352909 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:53.355314 (+ 2405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":94,"threads_started":1}
W20260928 16:39:53.355621 2641 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:53.348730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:49023 (local address 127.25.71.126:44945)
0928 16:39:53.349009 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:53.349013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:53.349088 (+ 75us) server_negotiation.cc:407] Connection header received
0928 16:39:53.349266 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:53.349270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:53.349334 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:53.349459 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:53.350110 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:53.350824 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:53.351841 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:53.352122 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:53.352225 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:53.355498 (+ 3273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.65:49023: BlockingRecv error: recv got EOF from 127.25.71.65:49023 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":68,"threads_started":1}
Sep 28 16:39:53 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:53 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:53.575903 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:53.568759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.66:52221)
0928 16:39:53.569290 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:53.569311 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:53.569535 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:53.569855 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:53.569858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:53.570136 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:53.570299 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:53.570308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:53.571335 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:53.571339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:53.572036 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:53.572047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:53.572531 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:53.572534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:53.572618 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:53.573289 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:53.573305 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:53.575766 (+ 2461us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":182,"threads_started":1}
W20260928 16:39:53.576238 2641 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:53.569061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:52221 (local address 127.25.71.126:44945)
0928 16:39:53.569280 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:53.569287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:53.569448 (+ 161us) server_negotiation.cc:407] Connection header received
0928 16:39:53.569650 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:53.569653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:53.569719 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:53.569797 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:53.570430 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:53.571221 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:53.572170 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:53.572413 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:53.572506 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:53.576067 (+ 3561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.66:52221: BlockingRecv error: recv got EOF from 127.25.71.66:52221 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 28 16:39:53 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:53 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:53.783493 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:53.777301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.67:37055)
0928 16:39:53.777631 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:53.777645 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:53.777762 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:53.778065 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:53.778067 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:53.778261 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:53.778425 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:53.778432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:53.779419 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:53.779422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:53.780037 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:53.780047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:53.780602 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:53.780605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:53.780684 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:53.781288 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:53.781300 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:53.783306 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":102,"threads_started":1}
W20260928 16:39:53.783699 2641 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:53.777400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:37055 (local address 127.25.71.126:44945)
0928 16:39:53.777550 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:53.777554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:53.777705 (+ 151us) server_negotiation.cc:407] Connection header received
0928 16:39:53.777859 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:53.777862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:53.777923 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:53.778016 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:53.778562 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:53.779189 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:53.780161 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:53.780451 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:53.780541 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:53.783530 (+ 2989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.67:37055: BlockingRecv error: recv got EOF from 127.25.71.67:37055 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 16:39:54 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:54 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:54.371387 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:54.356777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.65:46243)
0928 16:39:54.357126 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:54.357146 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:54.357297 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:54.357677 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:54.357681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:54.357952 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:54.358096 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:54.358104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:54.359107 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:54.359112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:54.359840 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:54.359853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:54.360575 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:54.360578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:54.360649 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:54.368752 (+ 8103us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:54.368776 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:54.371185 (+ 2409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":101,"threads_started":1}
W20260928 16:39:54.371630 2645 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:54.356909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:46243 (local address 127.25.71.126:44945)
0928 16:39:54.357358 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:54.357363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:54.357383 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:39:54.357444 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:54.357447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:54.357504 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:54.357614 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:54.358231 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:54.358921 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:54.360066 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:54.360445 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:54.360555 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:54.371456 (+ 10901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.65:46243: BlockingRecv error: recv got EOF from 127.25.71.65:46243 (error 108)
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":128,"threads_started":1}
Sep 28 16:39:54 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:54 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:54.583204 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:54.576909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.66:54431)
0928 16:39:54.577316 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:54.577337 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:54.577506 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:54.577801 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:54.577804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:54.578066 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:54.578190 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:54.578197 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:54.579137 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:54.579140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:54.579727 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:54.579736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:54.580219 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:54.580222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:54.580307 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:54.580892 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:54.580905 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:54.583013 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":107,"threads_started":1}
W20260928 16:39:54.583308 2645 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:54.577116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:54431 (local address 127.25.71.126:44945)
0928 16:39:54.577318 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:54.577323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:54.577438 (+ 115us) server_negotiation.cc:407] Connection header received
0928 16:39:54.577623 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:54.577626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:54.577679 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:54.577756 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:54.578318 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:54.578955 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:54.579846 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:54.580100 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:54.580192 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:54.583188 (+ 2996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.66:54431: BlockingRecv error: recv got EOF from 127.25.71.66:54431 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 28 16:39:54 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:54 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:54.791092 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:54.784543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.67:44517)
0928 16:39:54.785010 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:54.785024 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:54.785126 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:54.785405 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:54.785406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:54.785582 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:54.785689 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:54.785695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:54.786554 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:54.786557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:54.787307 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:54.787353 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:54.787928 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:54.787931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:54.788009 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:54.788595 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:54.788607 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:54.790953 (+ 2346us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":105,"threads_started":1}
W20260928 16:39:54.791349 2645 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:54.784634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:44517 (local address 127.25.71.126:44945)
0928 16:39:54.784807 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:54.784812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:54.785100 (+ 288us) server_negotiation.cc:407] Connection header received
0928 16:39:54.785211 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:54.785213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:54.785277 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:54.785358 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:54.785845 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:54.786446 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:54.787475 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:54.787767 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:54.787871 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:54.791187 (+ 3316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.67:44517: BlockingRecv error: recv got EOF from 127.25.71.67:44517 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 16:39:55 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:55 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:55.379630 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:55.372589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.65:49373)
0928 16:39:55.372931 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:55.372948 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:55.373072 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:55.373364 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:55.373366 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:55.373592 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:55.373737 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:55.373744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:55.374733 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:55.374736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:55.375428 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:55.375439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:55.376061 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:55.376064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:55.376225 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:55.376928 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:55.376942 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:55.379462 (+ 2520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":106,"threads_started":1}
W20260928 16:39:55.379807 2649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:55.372756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:49373 (local address 127.25.71.126:44945)
0928 16:39:55.373062 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:55.373065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:55.373085 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:39:55.373177 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:55.373179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:55.373226 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:55.373312 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:55.373902 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:55.374500 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:55.375589 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:55.375905 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:55.376044 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:55.379666 (+ 3622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.65:49373: BlockingRecv error: recv got EOF from 127.25.71.65:49373 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":95,"threads_started":1}
Sep 28 16:39:55 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:55 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:55.591092 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:55.584737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.66:51283)
0928 16:39:55.585071 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:55.585088 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:55.585208 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:55.585547 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:55.585550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:55.585762 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:55.585938 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:55.585946 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:55.586806 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:55.586817 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:55.587573 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:55.587584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:55.588070 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:55.588074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:55.588149 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:55.588879 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:55.588893 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:55.590888 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":116,"threads_started":1}
W20260928 16:39:55.591146 2649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:55.585074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:51283 (local address 127.25.71.126:44945)
0928 16:39:55.585288 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:55.585294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:55.585335 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:39:55.585383 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:55.585385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:55.585427 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:55.585495 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:55.586092 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:55.586700 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:55.587688 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:55.587956 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:55.588065 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:55.591044 (+ 2979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.66:51283: BlockingRecv error: recv got EOF from 127.25.71.66:51283 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 28 16:39:55 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:55 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:55.799300 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:55.792079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.67:38209)
0928 16:39:55.792637 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:55.792675 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:55.792811 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:55.793220 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:55.793223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:55.793542 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:55.793691 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:55.793700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:55.794582 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:55.794585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:55.795256 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:55.795267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:55.795781 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:55.795784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:55.795871 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:55.796482 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:55.796495 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:55.799082 (+ 2587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":182,"threads_started":1}
W20260928 16:39:55.799433 2649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:55.792133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:38209 (local address 127.25.71.126:44945)
0928 16:39:55.792297 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:55.792302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:55.792776 (+ 474us) server_negotiation.cc:407] Connection header received
0928 16:39:55.792932 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:55.792935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:55.792998 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:55.793078 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:55.793853 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:55.794459 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:55.795382 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:55.795655 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:55.795755 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:55.799313 (+ 3558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.67:38209: BlockingRecv error: recv got EOF from 127.25.71.67:38209 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 16:39:56 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:56 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:56.387514 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:56.380759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.65:48819)
0928 16:39:56.381117 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:56.381131 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:56.381227 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:56.381596 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:56.381599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:56.381794 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:56.381980 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:56.381986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:56.383060 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:56.383063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:56.383672 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:56.383682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:56.384327 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:56.384331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:56.384407 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:56.385028 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:56.385040 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:56.387374 (+ 2334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":104,"threads_started":1}
W20260928 16:39:56.387869 2653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:56.380885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:48819 (local address 127.25.71.126:44945)
0928 16:39:56.381264 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:56.381269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:56.381285 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:39:56.381333 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:56.381335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:56.381394 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:56.381506 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:56.382222 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:56.382910 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:56.383875 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:56.384182 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:56.384297 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:56.387678 (+ 3381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.65:48819: BlockingRecv error: recv got EOF from 127.25.71.65:48819 (error 108)
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":166,"threads_started":1}
Sep 28 16:39:56 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:56 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:56.598539 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:56.592189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.66:33465)
0928 16:39:56.592574 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:56.592593 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:56.592752 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:56.593094 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:56.593098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:56.593382 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:56.593522 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:56.593529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:56.594480 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:56.594484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:56.595127 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:56.595137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:56.595605 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:56.595608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:56.595690 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:56.596317 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:56.596331 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:56.598334 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":112,"threads_started":1}
W20260928 16:39:56.598711 2653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:56.592207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:33465 (local address 127.25.71.126:44945)
0928 16:39:56.592375 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:56.592380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:56.592772 (+ 392us) server_negotiation.cc:407] Connection header received
0928 16:39:56.592859 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:56.592862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:56.592933 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:56.593037 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:56.593663 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:56.594362 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:56.595252 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:56.595494 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:56.595585 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:56.598556 (+ 2971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.66:33465: BlockingRecv error: recv got EOF from 127.25.71.66:33465 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 16:39:56 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:56 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:56.807611 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:56.800464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.67:38595)
0928 16:39:56.800895 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:56.800916 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:56.801085 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:56.801349 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:56.801352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:56.801571 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:56.801702 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:56.801712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:56.802867 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:56.802926 (+ 59us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:56.803543 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:56.803555 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:56.804162 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:56.804165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:56.804257 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:56.804916 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:56.804930 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:56.807476 (+ 2546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":141,"threads_started":1}
W20260928 16:39:56.808000 2653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:56.800649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:38595 (local address 127.25.71.126:44945)
0928 16:39:56.800919 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:56.800925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:56.801022 (+ 97us) server_negotiation.cc:407] Connection header received
0928 16:39:56.801184 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:56.801187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:56.801237 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:56.801306 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:56.801895 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:56.802536 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:56.803727 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:56.804001 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:56.804109 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:56.807829 (+ 3720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.67:38595: BlockingRecv error: recv got EOF from 127.25.71.67:38595 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 28 16:39:57 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:57 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:57.396026 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:57.388640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.65:56699)
0928 16:39:57.389099 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:57.389119 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:57.389264 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:57.389628 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:57.389631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:57.389896 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:57.390119 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:57.390128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:57.391162 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:57.391166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:57.391756 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:57.391766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:57.392534 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:57.392537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:57.392621 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:57.393471 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:57.393486 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:57.395877 (+ 2391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":91,"threads_started":1}
W20260928 16:39:57.396426 2657 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:57.388814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:56699 (local address 127.25.71.126:44945)
0928 16:39:57.389079 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:57.389083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:57.389283 (+ 200us) server_negotiation.cc:407] Connection header received
0928 16:39:57.389359 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:57.389362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:57.389437 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:57.389546 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:57.390288 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:57.390995 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:57.392020 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:57.392356 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:57.392472 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:57.396234 (+ 3762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.65:56699: BlockingRecv error: recv got EOF from 127.25.71.65:56699 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":67,"threads_started":1}
Sep 28 16:39:57 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:57 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:57.606892 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:57.599757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.66:49393)
0928 16:39:57.600281 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:57.600297 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:57.600474 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:57.600739 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:57.600741 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:57.600957 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:57.601062 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:57.601069 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:57.601907 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:57.601910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:57.602512 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:57.602521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:57.603049 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:57.603053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:57.603140 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:57.603823 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:57.603838 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:57.606720 (+ 2882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":119,"threads_started":1}
W20260928 16:39:57.607110 2657 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:57.599940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:49393 (local address 127.25.71.126:44945)
0928 16:39:57.600212 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:57.600218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:57.600391 (+ 173us) server_negotiation.cc:407] Connection header received
0928 16:39:57.600577 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:57.600580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:57.600627 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:57.600694 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:57.601188 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:57.601794 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:57.602661 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:57.602917 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:57.603018 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:57.606960 (+ 3942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.66:49393: BlockingRecv error: recv got EOF from 127.25.71.66:49393 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
Sep 28 16:39:57 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:57 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:57.815349 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:57.808540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.67:57115)
0928 16:39:57.808992 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:57.809012 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:57.809149 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:57.809409 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:57.809412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:57.809677 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:57.809792 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:57.809799 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:57.810808 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:57.810813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:57.811561 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:57.811578 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:57.812121 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:57.812125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:57.812227 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:57.812857 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:57.812870 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:57.815205 (+ 2335us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":90,"threads_started":1}
W20260928 16:39:57.815716 2657 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:57.808666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:57115 (local address 127.25.71.126:44945)
0928 16:39:57.808844 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:57.808849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:57.809096 (+ 247us) server_negotiation.cc:407] Connection header received
0928 16:39:57.809242 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:57.809245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:57.809302 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:57.809370 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:57.809920 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:57.810593 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:57.811722 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:57.811984 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:57.812085 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:57.815508 (+ 3423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.67:57115: BlockingRecv error: recv got EOF from 127.25.71.67:57115 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 16:39:58 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:58 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:58.403836 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:58.397233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.65:53125)
0928 16:39:58.397511 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:58.397525 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:58.397629 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:58.398182 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:58.398185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:58.398384 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:58.398503 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:58.398509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:58.399453 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:58.399456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:58.400066 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:58.400076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:58.400596 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:58.400599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:58.400673 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:58.401359 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:58.401377 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:58.403705 (+ 2328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
W20260928 16:39:58.404251 2661 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:58.397556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:53125 (local address 127.25.71.126:44945)
0928 16:39:58.397850 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:58.397855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:58.397871 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:39:58.397928 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:58.397931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:58.397983 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:58.398097 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:58.398650 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:58.399310 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:58.400209 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:58.400464 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:58.400548 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:58.404002 (+ 3454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.65:53125: BlockingRecv error: recv got EOF from 127.25.71.65:53125 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":129,"threads_started":1}
Sep 28 16:39:58 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:58 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:58.615188 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:58.608011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.66:46869)
0928 16:39:58.608371 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:58.608390 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:58.608564 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:58.608898 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:58.608901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:58.609141 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:58.609287 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:58.609296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:58.610578 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:58.610583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:58.611363 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:58.611378 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:58.611925 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:58.611928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:58.612013 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:58.612619 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:58.612633 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:58.614966 (+ 2333us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":79,"threads_started":1}
W20260928 16:39:58.615651 2661 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:58.608207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:46869 (local address 127.25.71.126:44945)
0928 16:39:58.608418 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:58.608423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:58.608505 (+ 82us) server_negotiation.cc:407] Connection header received
0928 16:39:58.608718 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:58.608722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:58.608779 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:58.608853 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:58.609491 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:58.610419 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:58.611504 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:58.611766 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:58.611885 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:58.615495 (+ 3610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.66:46869: BlockingRecv error: recv got EOF from 127.25.71.66:46869 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 16:39:58 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:58 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:58.822798 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:58.816465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.67:49441)
0928 16:39:58.816809 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:58.816836 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:58.816967 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:58.817290 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:58.817294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:58.817526 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:58.817671 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:58.817680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:58.818610 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:58.818613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:58.819392 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:58.819404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:58.820043 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:58.820046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:58.820132 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:58.820706 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:58.820719 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:58.822610 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":116,"threads_started":1}
W20260928 16:39:58.823305 2661 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:58.816490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:49441 (local address 127.25.71.126:44945)
0928 16:39:58.816641 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:58.816647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:58.816923 (+ 276us) server_negotiation.cc:407] Connection header received
0928 16:39:58.817065 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:58.817068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:58.817145 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:58.817248 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:58.817802 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:58.818452 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:58.819557 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:58.819854 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:58.819967 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:58.823148 (+ 3181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.67:49441: BlockingRecv error: recv got EOF from 127.25.71.67:49441 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 16:39:59 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:59 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:59.415570 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:59.404851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.65:48861)
0928 16:39:59.405140 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:59.405154 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:59.405247 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:59.405693 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:59.405695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:59.405937 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:59.406061 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:59.406066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:59.407072 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:59.407075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:59.407656 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:59.407665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:59.408261 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:59.408264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:59.408338 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:59.412563 (+ 4225us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:59.412580 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:59.415376 (+ 2796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":103,"threads_started":1}
W20260928 16:39:59.415853 2665 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:59.405034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:48861 (local address 127.25.71.126:44945)
0928 16:39:59.405381 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:59.405385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:59.405401 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:39:59.405450 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:59.405452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:59.405506 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:59.405642 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:59.406217 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:59.406930 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:59.407821 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:59.408115 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:59.408221 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:59.415667 (+ 7446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.65:48861: BlockingRecv error: recv got EOF from 127.25.71.65:48861 (error 108)
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":96,"threads_started":1}
Sep 28 16:39:59 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:59 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:59.623257 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:59.616448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.66:49141)
0928 16:39:59.616975 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:59.616994 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:59.617152 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:59.617507 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:59.617510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:59.617709 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:59.617844 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:59.617851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:59.618925 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:59.618929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:59.619583 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:59.619594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:59.620113 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:59.620116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:59.620192 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:59.620784 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:59.620797 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:59.623118 (+ 2321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":142,"threads_started":1}
W20260928 16:39:59.623577 2665 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:59.616570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:49141 (local address 127.25.71.126:44945)
0928 16:39:59.616834 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:59.616840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:59.617115 (+ 275us) server_negotiation.cc:407] Connection header received
0928 16:39:59.617274 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:59.617278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:59.617342 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:59.617434 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:59.617970 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:59.618764 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:59.619715 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:59.619975 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:59.620078 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:59.623409 (+ 3331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.66:49141: BlockingRecv error: recv got EOF from 127.25.71.66:49141 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Sep 28 16:39:59 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:39:59 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:39:59.833941 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:59.823730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.67:53973)
0928 16:39:59.824174 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:59.824191 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:59.824302 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:59.824570 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:59.824573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:59.824813 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:59.824913 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:59.824919 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:59.825686 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:59.825689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:59.826262 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:59.826270 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:59.826738 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:59.826742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:59.826823 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:59.831103 (+ 4280us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:59.831121 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:59.833772 (+ 2651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":124,"threads_started":1}
W20260928 16:39:59.834192 2665 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:59.824035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:53973 (local address 127.25.71.126:44945)
0928 16:39:59.824229 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:59.824235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:59.824252 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:39:59.824401 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:59.824404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:59.824460 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:59.824536 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:59.825029 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:59.825580 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:59.826376 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:59.826599 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:59.826711 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:59.834042 (+ 7331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.67:53973: BlockingRecv error: recv got EOF from 127.25.71.67:53973 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 28 16:40:00 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:40:00 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:40:00.423944 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:00.416685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.65:41999)
0928 16:40:00.417075 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:00.417094 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:00.417225 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:00.417610 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:00.417612 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:00.417809 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:00.417923 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:00.417929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:00.418941 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:00.418945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:00.419555 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:00.419566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:00.420135 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:00.420138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:00.420215 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:00.421021 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:40:00.421035 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:00.423698 (+ 2663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":122,"threads_started":1}
W20260928 16:40:00.424073 2669 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:40:00.416808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:41999 (local address 127.25.71.126:44945)
0928 16:40:00.417118 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:00.417122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:00.417299 (+ 177us) server_negotiation.cc:407] Connection header received
0928 16:40:00.417362 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:00.417364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:00.417427 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:00.417516 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:00.418075 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:00.418807 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:00.419699 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:00.419972 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:00.420064 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:00.423926 (+ 3862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.65:41999: BlockingRecv error: recv got EOF from 127.25.71.65:41999 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":72,"threads_started":1}
Sep 28 16:40:00 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:40:00 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:40:00.631732 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:00.624586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.66:57003)
0928 16:40:00.625091 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:00.625114 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:00.625284 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:00.625602 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:00.625606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:00.625945 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:00.626095 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:00.626104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:00.627073 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:00.627077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:00.627694 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:00.627705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:00.628193 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:00.628197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:00.628281 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:00.628973 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:40:00.629075 (+ 102us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:00.631543 (+ 2468us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":100,"threads_started":1}
W20260928 16:40:00.631870 2669 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:40:00.624787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:57003 (local address 127.25.71.126:44945)
0928 16:40:00.625011 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:00.625018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:00.625223 (+ 205us) server_negotiation.cc:407] Connection header received
0928 16:40:00.625398 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:00.625402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:00.625462 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:00.625560 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:00.626243 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:00.626946 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:00.627820 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:00.628061 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:00.628168 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:00.631756 (+ 3588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.66:57003: BlockingRecv error: recv got EOF from 127.25.71.66:57003 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 28 16:40:00 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:40:00 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:40:00.842657 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:00.835249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.67:55017)
0928 16:40:00.835725 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:00.835746 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:00.835977 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:00.836258 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:00.836261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:00.836480 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:00.836581 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:00.836587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:00.837496 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:00.837501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:00.838426 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:00.838445 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:00.839197 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:00.839201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:00.839293 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:00.839943 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:40:00.839957 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:00.842406 (+ 2449us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":119,"threads_started":1}
W20260928 16:40:00.843014 2669 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:40:00.835522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:55017 (local address 127.25.71.126:44945)
0928 16:40:00.835729 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:00.835735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:00.835914 (+ 179us) server_negotiation.cc:407] Connection header received
0928 16:40:00.836081 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:00.836084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:00.836139 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:00.836210 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:00.836702 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:00.837273 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:00.838586 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:00.839022 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:00.839136 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:00.842767 (+ 3631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.67:55017: BlockingRecv error: recv got EOF from 127.25.71.67:55017 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 28 16:40:01 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:40:01 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:40:01.431878 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:01.425282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.65:57169)
0928 16:40:01.425877 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:01.425897 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:01.426033 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:01.426418 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:01.426421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:01.426617 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:01.426773 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:01.426780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:01.427872 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:01.427876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:01.428627 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:01.428642 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:01.429187 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:01.429190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:01.429262 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:01.429846 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:40:01.429859 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:01.431737 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":178,"threads_started":1}
W20260928 16:40:01.432299 2673 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:40:01.425516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:57169 (local address 127.25.71.126:44945)
0928 16:40:01.425920 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:01.425924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:01.426056 (+ 132us) server_negotiation.cc:407] Connection header received
0928 16:40:01.426132 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:01.426135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:01.426207 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:01.426370 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:01.426962 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:01.427713 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:01.428790 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:01.429064 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:01.429159 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:01.432087 (+ 2928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.65:57169: BlockingRecv error: recv got EOF from 127.25.71.65:57169 (error 108)
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":189,"threads_started":1}
Sep 28 16:40:01 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:40:01 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:40:01.639755 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:01.632928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.66:58081)
0928 16:40:01.633224 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:01.633239 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:01.633375 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:01.633689 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:01.633692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:01.633958 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:01.634186 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:01.634194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:01.635328 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:01.635331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:01.636035 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:01.636047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:01.636764 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:01.636767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:01.636850 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:01.637561 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:40:01.637574 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:01.639589 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":83,"threads_started":1}
W20260928 16:40:01.640044 2673 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:40:01.633035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:58081 (local address 127.25.71.126:44945)
0928 16:40:01.633244 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:01.633248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:01.633405 (+ 157us) server_negotiation.cc:407] Connection header received
0928 16:40:01.633485 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:01.633488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:01.633560 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:01.633655 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:01.634383 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:01.635145 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:01.636243 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:01.636532 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:01.636748 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:01.639864 (+ 3116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.66:58081: BlockingRecv error: recv got EOF from 127.25.71.66:58081 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 16:40:01 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:40:01 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:40:01.852475 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:01.843778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.67:38599)
0928 16:40:01.844230 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:01.844249 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:01.844455 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:01.845369 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:01.845372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:01.845829 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:01.845981 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:01.845998 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:01.847061 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:01.847065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:01.847691 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:01.847702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:01.848221 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:01.848225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:01.848314 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:01.849095 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:40:01.849112 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:01.852254 (+ 3142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":178,"threads_started":1}
W20260928 16:40:01.852607 2673 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:40:01.843943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:38599 (local address 127.25.71.126:44945)
0928 16:40:01.844136 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:01.844142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:01.844410 (+ 268us) server_negotiation.cc:407] Connection header received
0928 16:40:01.845115 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:01.845120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:01.845208 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:01.845328 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:01.846232 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:01.846926 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:01.847832 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:01.848093 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:01.848192 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:01.852463 (+ 4271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.67:38599: BlockingRecv error: recv got EOF from 127.25.71.67:38599 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 16:40:02 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:40:02 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:40:02.440651 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:02.433069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.65:52385)
0928 16:40:02.433436 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:02.433459 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:02.433590 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:02.434155 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:02.434159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:02.434411 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:02.434568 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:02.434574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:02.435648 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:02.435652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:02.436345 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:02.436357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:02.437122 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:02.437128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:02.437241 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:02.437983 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:40:02.437999 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:02.440482 (+ 2483us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":112,"threads_started":1}
W20260928 16:40:02.440778 2677 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:40:02.433213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:52385 (local address 127.25.71.126:44945)
0928 16:40:02.433657 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:02.433661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:02.433680 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:40:02.433735 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:02.433737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:02.433788 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:02.433996 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:02.434788 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:02.435440 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:02.436530 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:02.436854 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:02.436974 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:02.440649 (+ 3675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.65:52385: BlockingRecv error: recv got EOF from 127.25.71.65:52385 (error 108)
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":110,"threads_started":1}
Sep 28 16:40:02 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:40:02 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:40:02.647286 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:02.641098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.66:44139)
0928 16:40:02.641411 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:02.641429 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:02.641555 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:02.641864 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:02.641867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:02.642072 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:02.642171 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:02.642177 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:02.643154 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:02.643157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:02.643716 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:02.643725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:02.644221 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:02.644225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:02.644304 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:02.644890 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:40:02.644902 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:02.647148 (+ 2246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":87,"threads_started":1}
W20260928 16:40:02.647682 2677 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:40:02.641364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:44139 (local address 127.25.71.126:44945)
0928 16:40:02.641586 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:02.641591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:02.641621 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:40:02.641671 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:02.641673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:02.641744 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:02.641813 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:02.642296 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:02.643025 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:02.643835 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:02.644109 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:02.644199 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:02.647511 (+ 3312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.66:44139: BlockingRecv error: recv got EOF from 127.25.71.66:44139 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 28 16:40:02 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:40:02 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:40:02.860882 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:02.853794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.67:51337)
0928 16:40:02.854130 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:02.854147 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:02.854305 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:02.854660 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:02.854663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:02.854921 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:02.855080 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:02.855088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:02.856099 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:02.856103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:02.856860 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:02.856875 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:02.857507 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:02.857512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:02.857627 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:02.858285 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:40:02.858298 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:02.860697 (+ 2399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":149,"threads_started":1}
W20260928 16:40:02.861076 2677 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:40:02.853995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:51337 (local address 127.25.71.126:44945)
0928 16:40:02.854160 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:02.854165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:02.854251 (+ 86us) server_negotiation.cc:407] Connection header received
0928 16:40:02.854414 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:02.854417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:02.854503 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:02.854585 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:02.855251 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:02.855937 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:02.857011 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:02.857318 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:02.857436 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:02.860959 (+ 3523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.67:51337: BlockingRecv error: recv got EOF from 127.25.71.67:51337 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 16:40:03 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:40:03 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:40:03.448905 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:03.441828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.65:50281)
0928 16:40:03.442275 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:03.442299 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:03.442461 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:03.442877 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:03.442880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:03.443115 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:03.443258 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:03.443264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:03.444193 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:03.444197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:03.444893 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:03.444903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:03.445435 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:03.445438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:03.445518 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:03.446164 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:40:03.446181 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:03.448720 (+ 2539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":98,"threads_started":1}
W20260928 16:40:03.449090 2681 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:40:03.441989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:50281 (local address 127.25.71.126:44945)
0928 16:40:03.442279 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:03.442284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:03.442451 (+ 167us) server_negotiation.cc:407] Connection header received
0928 16:40:03.442581 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:03.442584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:03.442669 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:03.442854 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:03.443408 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:03.444059 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:03.445046 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:03.445293 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:03.445388 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:03.448915 (+ 3527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.65:50281: BlockingRecv error: recv got EOF from 127.25.71.65:50281 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":96,"threads_started":1}
Sep 28 16:40:03 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:40:03 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:40:03.654618 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:03.648337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.66:37659)
0928 16:40:03.648657 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:03.648673 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:03.648811 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:03.649095 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:03.649098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:03.649330 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:03.649442 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:03.649449 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:03.650363 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:03.650367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:03.651186 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:03.651202 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:03.651896 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:03.651900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:03.651999 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:03.652646 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:40:03.652659 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:03.654485 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":96,"threads_started":1}
W20260928 16:40:03.654917 2681 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:40:03.648637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:37659 (local address 127.25.71.126:44945)
0928 16:40:03.648848 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:03.648855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:03.648873 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:40:03.648930 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:03.648933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:03.648981 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:03.649052 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:03.649590 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:03.650178 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:03.651355 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:03.651730 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:03.651828 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:03.654749 (+ 2921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.66:37659: BlockingRecv error: recv got EOF from 127.25.71.66:37659 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 16:40:03 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:40:03 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:40:03.869203 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:03.862241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.67:52035)
0928 16:40:03.862584 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:03.862600 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:03.862831 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:03.863376 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:03.863380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:03.863581 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:03.863715 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:03.863722 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:03.864902 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:03.864907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:03.865613 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:03.865625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:03.866134 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:03.866138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:03.866212 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:03.866840 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:40:03.866851 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:03.869068 (+ 2217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":100,"threads_started":1}
W20260928 16:40:03.869524 2681 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:40:03.862375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:52035 (local address 127.25.71.126:44945)
0928 16:40:03.862698 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:03.862705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:03.862742 (+ 37us) server_negotiation.cc:407] Connection header received
0928 16:40:03.863029 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:03.863034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:03.863128 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:03.863292 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:03.863937 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:03.864715 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:03.865735 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:03.865988 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:03.866090 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:03.869357 (+ 3267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.67:52035: BlockingRecv error: recv got EOF from 127.25.71.67:52035 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 28 16:40:04 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:40:04 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:40:04.461719 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:04.450151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.65:59459)
0928 16:40:04.450495 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:04.450510 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:04.450616 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:04.451215 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:04.451217 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:04.451443 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:04.451571 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:04.451577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:04.452554 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:04.452557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:04.453218 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:04.453228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:04.453888 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:04.453893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:04.454008 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:04.459151 (+ 5143us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:40:04.459172 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:04.461549 (+ 2377us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":109,"threads_started":1}
W20260928 16:40:04.462136 2685 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:40:04.450449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:59459 (local address 127.25.71.126:44945)
0928 16:40:04.450893 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:04.450897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:04.450916 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:40:04.450973 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:04.450976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:04.451028 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:04.451158 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:04.451752 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:04.452406 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:04.453386 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:04.453667 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:04.453805 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:04.461864 (+ 8059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.65:59459: BlockingRecv error: recv got EOF from 127.25.71.65:59459 (error 108)
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":158,"threads_started":1}
Sep 28 16:40:04 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:40:04 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:40:04.662560 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:04.655746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.66:47787)
0928 16:40:04.656169 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:04.656188 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:04.656330 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:04.656772 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:04.656775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:04.657069 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:04.657202 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:04.657209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:04.658124 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:04.658127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:04.659013 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:04.659026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:04.659746 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:04.659751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:04.659907 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:04.660498 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:40:04.660511 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:04.662394 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":123,"threads_started":1}
W20260928 16:40:04.662755 2685 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:40:04.655880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:47787 (local address 127.25.71.126:44945)
0928 16:40:04.656064 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:04.656068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:04.656286 (+ 218us) server_negotiation.cc:407] Connection header received
0928 16:40:04.656502 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:04.656506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:04.656568 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:04.656690 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:04.657342 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:04.657945 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:04.659257 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:04.659568 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:04.659675 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:04.662586 (+ 2911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.66:47787: BlockingRecv error: recv got EOF from 127.25.71.66:47787 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 16:40:04 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:40:04 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.67@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:40:04.877461 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:04.870423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.67:35929)
0928 16:40:04.870886 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:04.870908 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:04.871112 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:04.871472 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:04.871475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:04.871742 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:04.871900 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:04.871909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:04.872903 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:04.872906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:04.873572 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:04.873581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:04.874136 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:04.874140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:04.874222 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:04.874912 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:40:04.874925 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:04.877271 (+ 2346us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":112,"threads_started":1}
W20260928 16:40:04.877595 2685 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:40:04.870686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:35929 (local address 127.25.71.126:44945)
0928 16:40:04.870845 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:04.870850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:04.871047 (+ 197us) server_negotiation.cc:407] Connection header received
0928 16:40:04.871230 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:04.871234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:04.871328 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:04.871408 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:04.872045 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:04.872785 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:04.873695 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:04.873968 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:04.874059 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:04.877458 (+ 3399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.67:35929: BlockingRecv error: recv got EOF from 127.25.71.67:35929 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 16:40:05 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:40:05 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.65@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:40:05.470038 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:05.463120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.65:50997)
0928 16:40:05.463639 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:05.463655 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:05.463784 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:05.464389 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:05.464392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:05.464617 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:05.464764 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:05.464771 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:05.465644 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:05.465648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:05.466287 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:05.466297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:05.466916 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:05.466920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:05.467003 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:05.467597 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:40:05.467609 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:05.469900 (+ 2291us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":120,"threads_started":1}
W20260928 16:40:05.470391 2689 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:40:05.463302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:50997 (local address 127.25.71.126:44945)
0928 16:40:05.463626 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:05.463631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:05.463741 (+ 110us) server_negotiation.cc:407] Connection header received
0928 16:40:05.463968 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:05.463973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:05.464051 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:05.464170 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:05.464896 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:05.465500 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:05.466438 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:05.466758 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:05.466852 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:05.470207 (+ 3355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.65:50997: BlockingRecv error: recv got EOF from 127.25.71.65:50997 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":89,"threads_started":1}
Sep 28 16:40:05 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:40:05 dist-test-slave-nqgh krb5kdc[1617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.71.66@KRBTEST.COM for oryx/127.25.71.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:40:05.671458 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:05.664237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:44945 (local address 127.25.71.66:51307)
0928 16:40:05.664552 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:05.664567 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:05.664670 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:05.664988 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:05.664990 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:05.665180 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:05.665300 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:05.665306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:05.666394 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:05.666397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:05.667138 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:05.667155 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:05.667874 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:05.667878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:05.667982 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:05.668728 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:40:05.668747 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:05.671215 (+ 2468us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.71.126:44945: Server oryx/127.25.71.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":101,"threads_started":1}
W20260928 16:40:05.671545 2689 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:40:05.664438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:51307 (local address 127.25.71.126:44945)
0928 16:40:05.664656 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:05.664661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:05.664684 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:40:05.664797 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:05.664800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:05.664855 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:05.664935 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:05.665489 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:05.666155 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:05.667342 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:05.667671 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:05.667810 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:05.671423 (+ 3613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.71.66:51307: BlockingRecv error: recv got EOF from 127.25.71.66:51307 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 16:40:05.741858 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 2211
I20260928 16:40:05.749068 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 2365
I20260928 16:40:05.755599 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 2498
I20260928 16:40:05.762542 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 2140
2026-09-28T16:40:05Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18349 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T16:40:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:40:05Z Disabled control of system clock
I20260928 16:40:05.799638 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:38277
--webserver_interface=127.25.71.126
--webserver_port=0
--builtin_ntp_servers=127.25.71.84:35407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:38277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 16:40:05.940438 2694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:40:05.940856 2694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:40:05.940912 2694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:40:05.945740 2694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:40:05.945868 2694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:40:05.945890 2694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:40:05.945914 2694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:40:05.945932 2694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:40:05.952067 2694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:35407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:38277
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:38277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:40:05.953716 2694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:40:05.955832 2694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:40:05.967309 2700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:40:05.967314 2699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:40:05.967442 2694 server_base.cc:1034] running on GCE node
W20260928 16:40:05.967316 2702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:40:05.968536 2694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:40:05.970027 2694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:40:05.971273 2694 hybrid_clock.cc:648] HybridClock initialized: now 1790613605971246 us; error 58 us; skew 500 ppm
I20260928 16:40:05.974210 2694 webserver.cc:460] Webserver started at http://127.25.71.126:38665/ using document root <none> and password file <none>
I20260928 16:40:05.975015 2694 fs_manager.cc:359] Metadata directory not provided
I20260928 16:40:05.975106 2694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:40:05.975348 2694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:40:05.978094 2694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "05d74dd1baa84694bc48bfb9f39f74c5"
format_stamp: "Formatted at 2026-09-28 16:40:05 on dist-test-slave-nqgh"
server_key: "2f34a19271a72f29874662266a1d83b4"
server_key_iv: "ea5f9fab412d5d823d9db88d2546899d"
server_key_version: "encryptionkey@0"
I20260928 16:40:05.978816 2694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "05d74dd1baa84694bc48bfb9f39f74c5"
format_stamp: "Formatted at 2026-09-28 16:40:05 on dist-test-slave-nqgh"
server_key: "2f34a19271a72f29874662266a1d83b4"
server_key_iv: "ea5f9fab412d5d823d9db88d2546899d"
server_key_version: "encryptionkey@0"
I20260928 16:40:05.983970 2694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 16:40:05.987515 2708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:05.989357 2694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:40:05.989662 2694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "05d74dd1baa84694bc48bfb9f39f74c5"
format_stamp: "Formatted at 2026-09-28 16:40:05 on dist-test-slave-nqgh"
server_key: "2f34a19271a72f29874662266a1d83b4"
server_key_iv: "ea5f9fab412d5d823d9db88d2546899d"
server_key_version: "encryptionkey@0"
I20260928 16:40:05.989861 2694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:40:06.002744 2694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:40:06.003551 2694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:40:06.003738 2694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:40:06.011468 2694 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:38277
I20260928 16:40:06.011516 2761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:38277 every 8 connection(s)
I20260928 16:40:06.012595 2694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:40:06.017292 2762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:40:06.018466 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 2694
I20260928 16:40:06.018586 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:40:06.018949 25885 external_mini_cluster.cc:1428] Setting key 051e8bb85b8d0503ad6c480c4037a99e
I20260928 16:40:06.026268 2762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5: Bootstrap starting.
I20260928 16:40:06.029927 2762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5: Neither blocks nor log segments found. Creating new log.
I20260928 16:40:06.031136 2762 log.cc:824] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5: Log is configured to *not* fsync() on all Append() calls
I20260928 16:40:06.031920 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:06.020799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34744 (local address 127.25.71.126:38277)
0928 16:40:06.021742 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:06.021768 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:06.021840 (+ 72us) server_negotiation.cc:407] Connection header received
0928 16:40:06.022892 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:06.022923 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:06.023562 (+ 639us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:06.024022 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:06.024596 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:06.026202 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:06.027314 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:06.028202 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:06.028437 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:06.029270 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:40:06.029277 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:40:06.029298 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:40:06.029927 (+ 629us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:40:06.030019 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:40:06.030051 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:40:06.030186 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:40:06.030370 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 16:40:06.030769 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":150,"threads_started":1}
I20260928 16:40:06.034019 2762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5: No bootstrap required, opened a new log
I20260928 16:40:06.038127 2762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d74dd1baa84694bc48bfb9f39f74c5" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 38277 } }
I20260928 16:40:06.038479 2762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:40:06.038550 2762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05d74dd1baa84694bc48bfb9f39f74c5, State: Initialized, Role: FOLLOWER
I20260928 16:40:06.039081 2762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d74dd1baa84694bc48bfb9f39f74c5" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 38277 } }
I20260928 16:40:06.039232 2762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:40:06.039304 2762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:40:06.039409 2762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:40:06.040647 2762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d74dd1baa84694bc48bfb9f39f74c5" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 38277 } }
I20260928 16:40:06.041087 2762 leader_election.cc:302] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05d74dd1baa84694bc48bfb9f39f74c5; no voters:
I20260928 16:40:06.041543 2762 leader_election.cc:288] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:40:06.041795 2767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:40:06.042141 2767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5 [term 1 LEADER]: Becoming Leader. State: Replica: 05d74dd1baa84694bc48bfb9f39f74c5, State: Running, Role: LEADER
I20260928 16:40:06.042610 2767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d74dd1baa84694bc48bfb9f39f74c5" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 38277 } }
I20260928 16:40:06.043080 2762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:40:06.044703 2768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05d74dd1baa84694bc48bfb9f39f74c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d74dd1baa84694bc48bfb9f39f74c5" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 38277 } } }
I20260928 16:40:06.044885 2768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5 [sys.catalog]: This master's current role is: LEADER
I20260928 16:40:06.045284 2769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05d74dd1baa84694bc48bfb9f39f74c5. Latest consensus state: current_term: 1 leader_uuid: "05d74dd1baa84694bc48bfb9f39f74c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d74dd1baa84694bc48bfb9f39f74c5" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 38277 } } }
I20260928 16:40:06.045379 2769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5 [sys.catalog]: This master's current role is: LEADER
W20260928 16:40:06.049515 2782 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:40:06.049655 2782 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:40:06.049773 2779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:40:06.050887 2779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:40:06.061556 2779 catalog_manager.cc:1347] Generated new cluster ID: 1e81045369a74b07a8b438a8c9c454ab
I20260928 16:40:06.061689 2779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:40:06.095013 2779 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:40:06.096508 2779 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:40:06.114001 2779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05d74dd1baa84694bc48bfb9f39f74c5: Generated new TSK 0
I20260928 16:40:06.115231 2779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:40:06.129943 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:38277
--builtin_ntp_servers=127.25.71.84:35407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 16:40:06.269517 2786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:40:06.269923 2786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:40:06.269968 2786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:40:06.274691 2786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:40:06.274804 2786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:40:06.274869 2786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:40:06.280627 2786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:35407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:38277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:40:06.282232 2786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:40:06.284174 2786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:40:06.296865 2792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:40:06.296795 2791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:40:06.296947 2786 server_base.cc:1034] running on GCE node
W20260928 16:40:06.296795 2794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:40:06.297854 2786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:40:06.298840 2786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:40:06.300055 2786 hybrid_clock.cc:648] HybridClock initialized: now 1790613606300019 us; error 52 us; skew 500 ppm
I20260928 16:40:06.302917 2786 webserver.cc:460] Webserver started at http://127.25.71.65:46109/ using document root <none> and password file <none>
I20260928 16:40:06.303712 2786 fs_manager.cc:359] Metadata directory not provided
I20260928 16:40:06.303797 2786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:40:06.304041 2786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:40:06.306854 2786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "28c0a1ceabab49158fe533a3f940c764"
format_stamp: "Formatted at 2026-09-28 16:40:06 on dist-test-slave-nqgh"
server_key: "4c02dbcbf18430f7e483d388fcd19a51"
server_key_iv: "d18766d2f5f680ab9539c33ae149c47c"
server_key_version: "encryptionkey@0"
I20260928 16:40:06.307564 2786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "28c0a1ceabab49158fe533a3f940c764"
format_stamp: "Formatted at 2026-09-28 16:40:06 on dist-test-slave-nqgh"
server_key: "4c02dbcbf18430f7e483d388fcd19a51"
server_key_iv: "d18766d2f5f680ab9539c33ae149c47c"
server_key_version: "encryptionkey@0"
I20260928 16:40:06.312644 2786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:40:06.316479 2800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:06.318490 2786 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 16:40:06.318727 2786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "28c0a1ceabab49158fe533a3f940c764"
format_stamp: "Formatted at 2026-09-28 16:40:06 on dist-test-slave-nqgh"
server_key: "4c02dbcbf18430f7e483d388fcd19a51"
server_key_iv: "d18766d2f5f680ab9539c33ae149c47c"
server_key_version: "encryptionkey@0"
I20260928 16:40:06.318908 2786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:40:06.334578 2786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:40:06.335634 2786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:40:06.335842 2786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:40:06.336611 2786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:40:06.337942 2786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:40:06.337999 2786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:06.338066 2786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:40:06.338092 2786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:06.350212 2786 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:36459
I20260928 16:40:06.350418 2914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:36459 every 8 connection(s)
I20260928 16:40:06.351435 2786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:40:06.359206 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 2786
I20260928 16:40:06.359336 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:40:06.359642 25885 external_mini_cluster.cc:1428] Setting key 6628f1e1dbae1addcea9f9a2d6fbb07b
I20260928 16:40:06.362263 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:38277
--builtin_ntp_servers=127.25.71.84:35407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 16:40:06.365108 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:06.354347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:49127 (local address 127.25.71.126:38277)
0928 16:40:06.354536 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:06.354541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:06.356274 (+ 1733us) server_negotiation.cc:407] Connection header received
0928 16:40:06.358029 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:06.358033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:06.358107 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:06.358216 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:06.359704 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:06.360532 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:06.362707 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:06.363066 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:06.363187 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:06.364569 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:40:06.364573 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:40:06.364575 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:40:06.364591 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:40:06.364618 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:40:06.364627 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:40:06.364687 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 16:40:06.364849 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 16:40:06.364946 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 16:40:06.366086 2917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:06.354940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:38277 (local address 127.25.71.65:49127)
0928 16:40:06.355999 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:06.356064 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:06.357662 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:06.358485 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:06.358494 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:06.358616 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:06.359490 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:06.359509 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:06.360807 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:06.360812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:06.362452 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:06.362476 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:06.363273 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:06.363277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:06.363515 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:06.364225 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:40:06.364227 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:06.364279 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:40:06.364366 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:40:06.364700 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:40:06.364710 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:40:06.364716 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:40:06.364845 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 16:40:06.365196 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":178,"threads_started":1}
I20260928 16:40:06.367702 2915 heartbeater.cc:347] Connected to a master server at 127.25.71.126:38277
I20260928 16:40:06.368171 2915 heartbeater.cc:464] Registering TS with master...
I20260928 16:40:06.369222 2915 heartbeater.cc:511] Master 127.25.71.126:38277 requested a full tablet report, sending...
I20260928 16:40:06.371637 2723 ts_manager.cc:194] Registered new tserver with Master: 28c0a1ceabab49158fe533a3f940c764 (127.25.71.65:36459)
I20260928 16:40:06.372954 2723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.71.65:49127
W20260928 16:40:06.502180 2918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:40:06.502584 2918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:40:06.502664 2918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:40:06.507479 2918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:40:06.507577 2918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:40:06.507644 2918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:40:06.513259 2918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:35407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:38277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:40:06.514961 2918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:40:06.516795 2918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:40:06.529517 2923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:40:06.529567 2924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:40:06.529923 2918 server_base.cc:1034] running on GCE node
W20260928 16:40:06.529578 2926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:40:06.530616 2918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:40:06.531490 2918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:40:06.532689 2918 hybrid_clock.cc:648] HybridClock initialized: now 1790613606532665 us; error 45 us; skew 500 ppm
I20260928 16:40:06.534890 2918 webserver.cc:460] Webserver started at http://127.25.71.66:36245/ using document root <none> and password file <none>
I20260928 16:40:06.535584 2918 fs_manager.cc:359] Metadata directory not provided
I20260928 16:40:06.535661 2918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:40:06.535878 2918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:40:06.538190 2918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "a20263f8862747ee894b0e4b4805834a"
format_stamp: "Formatted at 2026-09-28 16:40:06 on dist-test-slave-nqgh"
server_key: "521d9dd11775bed6b7b753bce237187f"
server_key_iv: "50eb96d91c7939eeedbfb20392a19c2b"
server_key_version: "encryptionkey@0"
I20260928 16:40:06.538889 2918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "a20263f8862747ee894b0e4b4805834a"
format_stamp: "Formatted at 2026-09-28 16:40:06 on dist-test-slave-nqgh"
server_key: "521d9dd11775bed6b7b753bce237187f"
server_key_iv: "50eb96d91c7939eeedbfb20392a19c2b"
server_key_version: "encryptionkey@0"
I20260928 16:40:06.543859 2918 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 16:40:06.547722 2932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:06.549466 2918 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:40:06.549654 2918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "a20263f8862747ee894b0e4b4805834a"
format_stamp: "Formatted at 2026-09-28 16:40:06 on dist-test-slave-nqgh"
server_key: "521d9dd11775bed6b7b753bce237187f"
server_key_iv: "50eb96d91c7939eeedbfb20392a19c2b"
server_key_version: "encryptionkey@0"
I20260928 16:40:06.549793 2918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:40:06.569299 2918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:40:06.570075 2918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:40:06.570272 2918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:40:06.570964 2918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:40:06.572093 2918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:40:06.572147 2918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:06.572207 2918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:40:06.572234 2918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:06.582650 2918 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:39085
I20260928 16:40:06.582783 3046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:39085 every 8 connection(s)
I20260928 16:40:06.583691 2918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:40:06.591614 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 2918
I20260928 16:40:06.591763 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:40:06.592051 25885 external_mini_cluster.cc:1428] Setting key 7837b7fb3d5f94fc9d9d7996c81d3255
I20260928 16:40:06.594771 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:38277
--builtin_ntp_servers=127.25.71.84:35407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 16:40:06.596141 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:06.586172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:49935 (local address 127.25.71.126:38277)
0928 16:40:06.586372 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:06.586378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:06.588043 (+ 1665us) server_negotiation.cc:407] Connection header received
0928 16:40:06.589828 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:06.589832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:06.589916 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:06.590028 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:06.591280 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:06.591973 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:06.593565 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:06.593946 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:06.594056 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:06.595572 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:40:06.595576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:40:06.595578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:40:06.595595 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:40:06.595623 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:40:06.595631 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:40:06.595717 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 16:40:06.595917 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 16:40:06.595990 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:40:06.597239 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:06.586789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:38277 (local address 127.25.71.66:49935)
0928 16:40:06.587812 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:06.587871 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:06.589508 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:06.590252 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:06.590260 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:06.590396 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:06.591037 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:06.591052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:06.592154 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:06.592158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:06.593414 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:06.593426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:06.594140 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:06.594144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:06.594385 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:06.595209 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:40:06.595211 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:06.595274 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:40:06.595377 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:40:06.595737 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:40:06.595744 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:40:06.595750 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:40:06.595889 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 16:40:06.596237 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":159,"threads_started":1}
I20260928 16:40:06.598553 3047 heartbeater.cc:347] Connected to a master server at 127.25.71.126:38277
I20260928 16:40:06.598989 3047 heartbeater.cc:464] Registering TS with master...
I20260928 16:40:06.599905 3047 heartbeater.cc:511] Master 127.25.71.126:38277 requested a full tablet report, sending...
I20260928 16:40:06.601554 2723 ts_manager.cc:194] Registered new tserver with Master: a20263f8862747ee894b0e4b4805834a (127.25.71.66:39085)
I20260928 16:40:06.602525 2723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.71.66:49935
W20260928 16:40:06.731679 3050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:40:06.732071 3050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:40:06.732117 3050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:40:06.736707 3050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:40:06.736811 3050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:40:06.736877 3050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:40:06.742475 3050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:35407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:38277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:40:06.744194 3050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:40:06.746174 3050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:40:06.759037 3055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:40:06.759238 3056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:40:06.759029 3058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:40:06.759439 3050 server_base.cc:1034] running on GCE node
I20260928 16:40:06.760139 3050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:40:06.760982 3050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:40:06.762184 3050 hybrid_clock.cc:648] HybridClock initialized: now 1790613606762160 us; error 67 us; skew 500 ppm
I20260928 16:40:06.764503 3050 webserver.cc:460] Webserver started at http://127.25.71.67:37303/ using document root <none> and password file <none>
I20260928 16:40:06.765159 3050 fs_manager.cc:359] Metadata directory not provided
I20260928 16:40:06.765235 3050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:40:06.765473 3050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:40:06.767807 3050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "07eaad48d10b4866a5a234809743f9dc"
format_stamp: "Formatted at 2026-09-28 16:40:06 on dist-test-slave-nqgh"
server_key: "41bf921b582c932971e0432599e2dcc2"
server_key_iv: "90decb6a9f3d05d6df73b9f57ed1f9a4"
server_key_version: "encryptionkey@0"
I20260928 16:40:06.768442 3050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "07eaad48d10b4866a5a234809743f9dc"
format_stamp: "Formatted at 2026-09-28 16:40:06 on dist-test-slave-nqgh"
server_key: "41bf921b582c932971e0432599e2dcc2"
server_key_iv: "90decb6a9f3d05d6df73b9f57ed1f9a4"
server_key_version: "encryptionkey@0"
I20260928 16:40:06.773242 3050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 16:40:06.776732 3064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:06.778605 3050 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:40:06.778904 3050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "07eaad48d10b4866a5a234809743f9dc"
format_stamp: "Formatted at 2026-09-28 16:40:06 on dist-test-slave-nqgh"
server_key: "41bf921b582c932971e0432599e2dcc2"
server_key_iv: "90decb6a9f3d05d6df73b9f57ed1f9a4"
server_key_version: "encryptionkey@0"
I20260928 16:40:06.779060 3050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:40:06.801477 3050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:40:06.802408 3050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:40:06.802703 3050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:40:06.803429 3050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:40:06.804569 3050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:40:06.804625 3050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:06.804682 3050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:40:06.804709 3050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:06.816125 3050 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:33937
I20260928 16:40:06.816256 3178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:33937 every 8 connection(s)
I20260928 16:40:06.817203 3050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:40:06.824683 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 3050
I20260928 16:40:06.824833 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:40:06.825153 25885 external_mini_cluster.cc:1428] Setting key 6b95b8317206b9035bca690fb3c8f6e8
I20260928 16:40:06.831722 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:06.819790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:43257 (local address 127.25.71.126:38277)
0928 16:40:06.820205 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:06.820210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:06.822042 (+ 1832us) server_negotiation.cc:407] Connection header received
0928 16:40:06.824030 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:06.824034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:06.824119 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:06.824229 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:06.825803 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:06.826547 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:06.828586 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:06.828938 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:06.829066 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:06.830865 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:40:06.830870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:40:06.830874 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:40:06.830894 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:40:06.830925 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:40:06.830997 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:40:06.831106 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:40:06.831469 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 16:40:06.831555 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240}
I20260928 16:40:06.833104 3181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:06.820398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:38277 (local address 127.25.71.67:43257)
0928 16:40:06.821781 (+ 1383us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:06.821857 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:06.823691 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:06.824508 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:06.824519 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:06.824654 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:06.825525 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:06.825552 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:06.826747 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:06.826750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:06.828403 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:06.828421 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:06.829139 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:06.829142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:06.829359 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:06.830330 (+ 971us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:40:06.830332 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:06.830430 (+ 98us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:40:06.830549 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:40:06.831165 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:40:06.831175 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:40:06.831184 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 16:40:06.831419 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 16:40:06.831845 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":168,"threads_started":1}
I20260928 16:40:06.835004 3179 heartbeater.cc:347] Connected to a master server at 127.25.71.126:38277
I20260928 16:40:06.835367 3179 heartbeater.cc:464] Registering TS with master...
I20260928 16:40:06.836120 3179 heartbeater.cc:511] Master 127.25.71.126:38277 requested a full tablet report, sending...
I20260928 16:40:06.837749 2723 ts_manager.cc:194] Registered new tserver with Master: 07eaad48d10b4866a5a234809743f9dc (127.25.71.67:33937)
I20260928 16:40:06.838531 2723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.71.67:43257
I20260928 16:40:06.840993 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 16:40:06.844775 2765 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:40:06.843476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34746 (local address 127.25.71.126:38277)
0928 16:40:06.843636 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:06.843640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:06.843963 (+ 323us) server_negotiation.cc:407] Connection header received
0928 16:40:06.844079 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:06.844082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:06.844158 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:06.844304 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:06.844603 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34746: BlockingRecv error: recv got EOF from 127.0.0.1:34746 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:40:06.845201 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 2786
I20260928 16:40:06.852488 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 2918
I20260928 16:40:06.859192 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 3050
I20260928 16:40:06.864825 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 2694
2026-09-28T16:40:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1097 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T16:40:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:40:06Z Disabled control of system clock
I20260928 16:40:06.897984 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:41967
--webserver_interface=127.25.71.126
--webserver_port=0
--builtin_ntp_servers=127.25.71.84:45157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:41967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 16:40:07.034459 3190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:40:07.034936 3190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:40:07.034981 3190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:40:07.039765 3190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:40:07.039892 3190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:40:07.039906 3190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:40:07.039924 3190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:40:07.039939 3190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:40:07.045993 3190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:45157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:41967
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:41967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:40:07.047711 3190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:40:07.049715 3190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:40:07.061204 3198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:40:07.061256 3196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:40:07.061389 3195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:40:07.061553 3190 server_base.cc:1034] running on GCE node
I20260928 16:40:07.062484 3190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:40:07.064242 3190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:40:07.065459 3190 hybrid_clock.cc:648] HybridClock initialized: now 1790613607065430 us; error 56 us; skew 500 ppm
I20260928 16:40:07.068346 3190 webserver.cc:460] Webserver started at http://127.25.71.126:32935/ using document root <none> and password file <none>
I20260928 16:40:07.069370 3190 fs_manager.cc:359] Metadata directory not provided
I20260928 16:40:07.069505 3190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:40:07.069921 3190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:40:07.073336 3190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "18cf0a9229904926af1a6b4f49e13c93"
format_stamp: "Formatted at 2026-09-28 16:40:07 on dist-test-slave-nqgh"
server_key: "7556498b999a1862c29e6f95cc122f53"
server_key_iv: "71aa73b1dcda6104315bb06423fce3ce"
server_key_version: "encryptionkey@0"
I20260928 16:40:07.074203 3190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "18cf0a9229904926af1a6b4f49e13c93"
format_stamp: "Formatted at 2026-09-28 16:40:07 on dist-test-slave-nqgh"
server_key: "7556498b999a1862c29e6f95cc122f53"
server_key_iv: "71aa73b1dcda6104315bb06423fce3ce"
server_key_version: "encryptionkey@0"
I20260928 16:40:07.079797 3190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 16:40:07.084208 3204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:07.086323 3190 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260928 16:40:07.086541 3190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "18cf0a9229904926af1a6b4f49e13c93"
format_stamp: "Formatted at 2026-09-28 16:40:07 on dist-test-slave-nqgh"
server_key: "7556498b999a1862c29e6f95cc122f53"
server_key_iv: "71aa73b1dcda6104315bb06423fce3ce"
server_key_version: "encryptionkey@0"
I20260928 16:40:07.086796 3190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:40:07.093150 3190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:40:07.094049 3190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:40:07.094265 3190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:40:07.103533 3190 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:41967
I20260928 16:40:07.103600 3256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:41967 every 8 connection(s)
I20260928 16:40:07.104907 3190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:40:07.107049 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 3190
I20260928 16:40:07.107183 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:40:07.107509 25885 external_mini_cluster.cc:1428] Setting key 5f7c63a1b3b03248e8b445bfe6380579
I20260928 16:40:07.109517 3257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:40:07.115206 3260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:07.109444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36482 (local address 127.25.71.126:41967)
0928 16:40:07.110447 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:07.110476 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:07.110535 (+ 59us) server_negotiation.cc:407] Connection header received
0928 16:40:07.111437 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:07.111460 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:07.111959 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:07.112406 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:07.113443 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:40:07.113448 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:40:07.113465 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:40:07.113478 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:40:07.113555 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:40:07.113576 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:40:07.113756 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 16:40:07.113838 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 16:40:07.114163 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":183,"threads_started":1}
I20260928 16:40:07.116755 3257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93: Bootstrap starting.
I20260928 16:40:07.119452 3257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93: Neither blocks nor log segments found. Creating new log.
I20260928 16:40:07.120256 3257 log.cc:824] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93: Log is configured to *not* fsync() on all Append() calls
I20260928 16:40:07.122890 3257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93: No bootstrap required, opened a new log
I20260928 16:40:07.127177 3257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18cf0a9229904926af1a6b4f49e13c93" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41967 } }
I20260928 16:40:07.127472 3257 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:40:07.127521 3257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf0a9229904926af1a6b4f49e13c93, State: Initialized, Role: FOLLOWER
I20260928 16:40:07.127957 3257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18cf0a9229904926af1a6b4f49e13c93" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41967 } }
I20260928 16:40:07.128098 3257 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:40:07.128165 3257 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:40:07.128276 3257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:40:07.129323 3257 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18cf0a9229904926af1a6b4f49e13c93" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41967 } }
I20260928 16:40:07.129707 3257 leader_election.cc:302] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18cf0a9229904926af1a6b4f49e13c93; no voters:
I20260928 16:40:07.130043 3257 leader_election.cc:288] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:40:07.130260 3262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:40:07.130666 3262 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93 [term 1 LEADER]: Becoming Leader. State: Replica: 18cf0a9229904926af1a6b4f49e13c93, State: Running, Role: LEADER
I20260928 16:40:07.131184 3262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18cf0a9229904926af1a6b4f49e13c93" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41967 } }
I20260928 16:40:07.131460 3257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:40:07.133219 3263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18cf0a9229904926af1a6b4f49e13c93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18cf0a9229904926af1a6b4f49e13c93" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41967 } } }
I20260928 16:40:07.133282 3264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18cf0a9229904926af1a6b4f49e13c93. Latest consensus state: current_term: 1 leader_uuid: "18cf0a9229904926af1a6b4f49e13c93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18cf0a9229904926af1a6b4f49e13c93" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41967 } } }
I20260928 16:40:07.133479 3264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93 [sys.catalog]: This master's current role is: LEADER
I20260928 16:40:07.133406 3263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93 [sys.catalog]: This master's current role is: LEADER
I20260928 16:40:07.133891 3276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:40:07.139330 3276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:40:07.149196 3276 catalog_manager.cc:1347] Generated new cluster ID: 959e7e4dd8054c05947ce5936153a999
I20260928 16:40:07.149304 3276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:40:07.157968 3276 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:40:07.158557 3276 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:40:07.170580 3276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18cf0a9229904926af1a6b4f49e13c93: Generated new TSK 0
I20260928 16:40:07.172019 3276 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:40:07.179355 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41967
--builtin_ntp_servers=127.25.71.84:45157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 16:40:07.320084 3281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:40:07.320405 3281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:40:07.320464 3281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:40:07.324714 3281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:40:07.324813 3281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:40:07.324893 3281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:40:07.330742 3281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:45157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:40:07.332056 3281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:40:07.333616 3281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:40:07.345607 3290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:40:07.345752 3286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:40:07.345753 3288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:40:07.346004 3281 server_base.cc:1034] running on GCE node
I20260928 16:40:07.346766 3281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:40:07.347894 3281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:40:07.349103 3281 hybrid_clock.cc:648] HybridClock initialized: now 1790613607349071 us; error 48 us; skew 500 ppm
I20260928 16:40:07.351647 3281 webserver.cc:460] Webserver started at http://127.25.71.65:38369/ using document root <none> and password file <none>
I20260928 16:40:07.352417 3281 fs_manager.cc:359] Metadata directory not provided
I20260928 16:40:07.352502 3281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:40:07.352756 3281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:40:07.355525 3281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "debd684870d045aea4b6c2235d4b6899"
format_stamp: "Formatted at 2026-09-28 16:40:07 on dist-test-slave-nqgh"
server_key: "e8249f47d5b31f8ecf3f79cc005da4e2"
server_key_iv: "a092afa185c3d4a424131f96fe5541e2"
server_key_version: "encryptionkey@0"
I20260928 16:40:07.356300 3281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "debd684870d045aea4b6c2235d4b6899"
format_stamp: "Formatted at 2026-09-28 16:40:07 on dist-test-slave-nqgh"
server_key: "e8249f47d5b31f8ecf3f79cc005da4e2"
server_key_iv: "a092afa185c3d4a424131f96fe5541e2"
server_key_version: "encryptionkey@0"
I20260928 16:40:07.361529 3281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:40:07.365475 3296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:07.367908 3281 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:40:07.368170 3281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "debd684870d045aea4b6c2235d4b6899"
format_stamp: "Formatted at 2026-09-28 16:40:07 on dist-test-slave-nqgh"
server_key: "e8249f47d5b31f8ecf3f79cc005da4e2"
server_key_iv: "a092afa185c3d4a424131f96fe5541e2"
server_key_version: "encryptionkey@0"
I20260928 16:40:07.368366 3281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:40:07.375012 3281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:40:07.375902 3281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:40:07.376116 3281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:40:07.376820 3281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:40:07.378050 3281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:40:07.378309 3281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:07.378396 3281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:40:07.378423 3281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:07.390148 3281 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:40409
I20260928 16:40:07.390192 3409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:40409 every 8 connection(s)
I20260928 16:40:07.391480 3281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:40:07.397984 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 3281
I20260928 16:40:07.398137 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:40:07.398532 25885 external_mini_cluster.cc:1428] Setting key c20eb56dff9935a4e51553e62a778ec8
I20260928 16:40:07.400527 3260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:07.394204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:44925 (local address 127.25.71.126:41967)
0928 16:40:07.394394 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:07.394399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:07.395992 (+ 1593us) server_negotiation.cc:407] Connection header received
0928 16:40:07.397703 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:07.397707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:07.397761 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:07.397894 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:07.399842 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:40:07.399846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:40:07.399848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:40:07.399863 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:40:07.399889 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:40:07.399898 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:40:07.399994 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 16:40:07.400319 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 16:40:07.400375 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:40:07.401309 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41967
--builtin_ntp_servers=127.25.71.84:45157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 16:40:07.402066 3412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:07.394607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:41967 (local address 127.25.71.65:44925)
0928 16:40:07.395741 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:07.395807 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:07.397379 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:07.398202 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:07.398215 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:07.398382 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:07.399377 (+ 995us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:40:07.399379 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:07.399447 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:40:07.399571 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:40:07.400011 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:40:07.400025 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:40:07.400032 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 16:40:07.400314 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 16:40:07.400651 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":164,"threads_started":1}
I20260928 16:40:07.403717 3410 heartbeater.cc:347] Connected to a master server at 127.25.71.126:41967
I20260928 16:40:07.404017 3410 heartbeater.cc:464] Registering TS with master...
I20260928 16:40:07.404526 3410 heartbeater.cc:511] Master 127.25.71.126:41967 requested a full tablet report, sending...
I20260928 16:40:07.406432 3221 ts_manager.cc:194] Registered new tserver with Master: debd684870d045aea4b6c2235d4b6899 (127.25.71.65:40409)
W20260928 16:40:07.539640 3413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:40:07.540035 3413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:40:07.540081 3413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:40:07.544878 3413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:40:07.544992 3413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:40:07.545085 3413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:40:07.550954 3413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:45157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:40:07.552525 3413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:40:07.554443 3413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:40:07.567262 3418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:40:07.567262 3421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:40:07.567274 3419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:40:07.567477 3413 server_base.cc:1034] running on GCE node
I20260928 16:40:07.568254 3413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:40:07.569144 3413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:40:07.570359 3413 hybrid_clock.cc:648] HybridClock initialized: now 1790613607570327 us; error 50 us; skew 500 ppm
I20260928 16:40:07.572933 3413 webserver.cc:460] Webserver started at http://127.25.71.66:39797/ using document root <none> and password file <none>
I20260928 16:40:07.573745 3413 fs_manager.cc:359] Metadata directory not provided
I20260928 16:40:07.573824 3413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:40:07.574074 3413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:40:07.576903 3413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "292a23f9a6874acdb877a7e2c8239ba9"
format_stamp: "Formatted at 2026-09-28 16:40:07 on dist-test-slave-nqgh"
server_key: "dc4e46762a019c2e1c0496f55a8a8cf6"
server_key_iv: "ee81044d07a6d0c7164d888b795f009e"
server_key_version: "encryptionkey@0"
I20260928 16:40:07.577698 3413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "292a23f9a6874acdb877a7e2c8239ba9"
format_stamp: "Formatted at 2026-09-28 16:40:07 on dist-test-slave-nqgh"
server_key: "dc4e46762a019c2e1c0496f55a8a8cf6"
server_key_iv: "ee81044d07a6d0c7164d888b795f009e"
server_key_version: "encryptionkey@0"
I20260928 16:40:07.583067 3413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 16:40:07.586967 3427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:07.589178 3413 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:40:07.589401 3413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "292a23f9a6874acdb877a7e2c8239ba9"
format_stamp: "Formatted at 2026-09-28 16:40:07 on dist-test-slave-nqgh"
server_key: "dc4e46762a019c2e1c0496f55a8a8cf6"
server_key_iv: "ee81044d07a6d0c7164d888b795f009e"
server_key_version: "encryptionkey@0"
I20260928 16:40:07.589593 3413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:40:07.596297 3413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:40:07.597225 3413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:40:07.597469 3413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:40:07.598222 3413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:40:07.599619 3413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:40:07.599682 3413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:07.599752 3413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:40:07.599783 3413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:07.612553 3413 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:42699
I20260928 16:40:07.612651 3540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:42699 every 8 connection(s)
I20260928 16:40:07.613839 3413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:40:07.620630 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 3413
I20260928 16:40:07.620776 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:40:07.621340 25885 external_mini_cluster.cc:1428] Setting key f6646c5c002bb604362ebcdf70a0a6dc
I20260928 16:40:07.622556 3260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:07.616324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:57919 (local address 127.25.71.126:41967)
0928 16:40:07.616549 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:07.616555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:07.617973 (+ 1418us) server_negotiation.cc:407] Connection header received
0928 16:40:07.619705 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:07.619708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:07.619841 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:07.619960 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:07.621959 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:40:07.621963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:40:07.621965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:40:07.621985 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:40:07.622017 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:40:07.622028 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:40:07.622152 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 16:40:07.622368 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 16:40:07.622419 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 16:40:07.623790 3543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:07.616712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:41967 (local address 127.25.71.66:57919)
0928 16:40:07.617717 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:07.617780 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:07.619368 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:07.620272 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:07.620282 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:07.620426 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:07.621479 (+ 1053us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:40:07.621481 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:07.621578 (+ 97us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:40:07.621719 (+ 141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:40:07.622168 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:40:07.622194 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:40:07.622204 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 16:40:07.622428 (+ 224us) client_negotiation.cc:240] Negotiation successful
0928 16:40:07.622901 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":148,"threads_started":1}
I20260928 16:40:07.624238 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41967
--builtin_ntp_servers=127.25.71.84:45157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 16:40:07.625195 3541 heartbeater.cc:347] Connected to a master server at 127.25.71.126:41967
I20260928 16:40:07.625605 3541 heartbeater.cc:464] Registering TS with master...
I20260928 16:40:07.626305 3541 heartbeater.cc:511] Master 127.25.71.126:41967 requested a full tablet report, sending...
I20260928 16:40:07.627918 3221 ts_manager.cc:194] Registered new tserver with Master: 292a23f9a6874acdb877a7e2c8239ba9 (127.25.71.66:42699)
W20260928 16:40:07.750902 3544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:40:07.751212 3544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:40:07.751256 3544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:40:07.755512 3544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:40:07.755594 3544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:40:07.755656 3544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:40:07.760655 3544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:45157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:40:07.761853 3544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:40:07.763545 3544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:40:07.776952 3550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:40:07.777079 3552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:40:07.777066 3544 server_base.cc:1034] running on GCE node
W20260928 16:40:07.776959 3549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:40:07.778000 3544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:40:07.778930 3544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:40:07.780139 3544 hybrid_clock.cc:648] HybridClock initialized: now 1790613607780117 us; error 38 us; skew 500 ppm
I20260928 16:40:07.783116 3544 webserver.cc:460] Webserver started at http://127.25.71.67:42229/ using document root <none> and password file <none>
I20260928 16:40:07.784145 3544 fs_manager.cc:359] Metadata directory not provided
I20260928 16:40:07.784286 3544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:40:07.784592 3544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:40:07.787761 3544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "d2f3ebaefe4b41189871f4b8feb9f760"
format_stamp: "Formatted at 2026-09-28 16:40:07 on dist-test-slave-nqgh"
server_key: "c2502fecc5352ec088bcd29cd8ca8116"
server_key_iv: "6bf93389b10b168cb62a60c030acf55b"
server_key_version: "encryptionkey@0"
I20260928 16:40:07.788532 3544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "d2f3ebaefe4b41189871f4b8feb9f760"
format_stamp: "Formatted at 2026-09-28 16:40:07 on dist-test-slave-nqgh"
server_key: "c2502fecc5352ec088bcd29cd8ca8116"
server_key_iv: "6bf93389b10b168cb62a60c030acf55b"
server_key_version: "encryptionkey@0"
I20260928 16:40:07.794013 3544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:40:07.798385 3558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:07.800341 3544 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:40:07.800556 3544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "d2f3ebaefe4b41189871f4b8feb9f760"
format_stamp: "Formatted at 2026-09-28 16:40:07 on dist-test-slave-nqgh"
server_key: "c2502fecc5352ec088bcd29cd8ca8116"
server_key_iv: "6bf93389b10b168cb62a60c030acf55b"
server_key_version: "encryptionkey@0"
I20260928 16:40:07.800733 3544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:40:07.807085 3544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:40:07.807963 3544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:40:07.808166 3544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:40:07.808912 3544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:40:07.810230 3544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:40:07.810287 3544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:07.810352 3544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:40:07.810376 3544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:07.821022 3544 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:35597
I20260928 16:40:07.821128 3672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:35597 every 8 connection(s)
I20260928 16:40:07.822146 3544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:40:07.823307 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 3544
I20260928 16:40:07.823408 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:40:07.823704 25885 external_mini_cluster.cc:1428] Setting key e87a05c6ef1f04eaa296f8b6f2e0ab3c
I20260928 16:40:07.830511 3260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:07.824711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:41693 (local address 127.25.71.126:41967)
0928 16:40:07.824876 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:07.824880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:07.826465 (+ 1585us) server_negotiation.cc:407] Connection header received
0928 16:40:07.828120 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:07.828123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:07.828174 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:07.828306 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:07.829922 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:40:07.829933 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:40:07.829935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:40:07.829949 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:40:07.829986 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:40:07.829995 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:40:07.830111 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 16:40:07.830335 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 16:40:07.830389 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:40:07.831924 3675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:07.825175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:41967 (local address 127.25.71.67:41693)
0928 16:40:07.826247 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:07.826325 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:07.827819 (+ 1494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:07.828538 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:07.828547 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:07.828740 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:07.829529 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:40:07.829531 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:07.829592 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:40:07.829695 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:40:07.830097 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:40:07.830103 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:40:07.830108 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:40:07.830342 (+ 234us) client_negotiation.cc:240] Negotiation successful
0928 16:40:07.830697 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":145,"threads_started":1}
I20260928 16:40:07.833742 3673 heartbeater.cc:347] Connected to a master server at 127.25.71.126:41967
I20260928 16:40:07.834141 3673 heartbeater.cc:464] Registering TS with master...
I20260928 16:40:07.834794 3673 heartbeater.cc:511] Master 127.25.71.126:41967 requested a full tablet report, sending...
I20260928 16:40:07.836645 3221 ts_manager.cc:194] Registered new tserver with Master: d2f3ebaefe4b41189871f4b8feb9f760 (127.25.71.67:35597)
I20260928 16:40:07.839237 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 16:40:07.843227 3260 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:40:07.842055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36494 (local address 127.25.71.126:41967)
0928 16:40:07.842236 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:07.842240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:07.842296 (+ 56us) server_negotiation.cc:407] Connection header received
0928 16:40:07.842486 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:07.842489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:07.842537 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:07.842713 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:07.843098 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36494: BlockingRecv error: recv got EOF from 127.0.0.1:36494 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 16:40:07.843451 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 3281
I20260928 16:40:07.850126 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 3413
I20260928 16:40:07.855865 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 3544
I20260928 16:40:07.861617 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 3190
2026-09-28T16:40:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (997 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:40:07 dist-test-slave-nqgh krb5kdc[3683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:40:07 dist-test-slave-nqgh krb5kdc[3683](info): set up 2 sockets
Sep 28 16:40:07 dist-test-slave-nqgh krb5kdc[3683](info): commencing operation
krb5kdc: starting...
W20260928 16:40:08.845453 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:40:08 dist-test-slave-nqgh krb5kdc[3683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:40:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:40:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:40:08.992358 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:41521
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:41521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:40:09.127863 3696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:40:09.128285 3696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:40:09.128355 3696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:40:09.133050 3696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:40:09.133169 3696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:40:09.133200 3696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:40:09.133221 3696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:40:09.133250 3696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:40:09.140082 3696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:41521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:41521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:40:09.141758 3696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:40:09.143277 3696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:40:09.154681 3701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:40:09.154805 3704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:40:09.154825 3696 server_base.cc:1034] running on GCE node
W20260928 16:40:09.154681 3702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:40:09.155992 3696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:40:09.157593 3696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:40:09.158768 3696 hybrid_clock.cc:648] HybridClock initialized: now 1790613609158740 us; error 47 us; skew 500 ppm
Sep 28 16:40:09 dist-test-slave-nqgh krb5kdc[3683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613609, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:40:09.162549 3696 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:40:09.163885 3696 webserver.cc:460] Webserver started at http://127.25.71.126:42181/ using document root <none> and password file <none>
I20260928 16:40:09.164619 3696 fs_manager.cc:359] Metadata directory not provided
I20260928 16:40:09.164714 3696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:40:09.164968 3696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:40:09.167690 3696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "913581dad71b4327a1dba1cb8560b317"
format_stamp: "Formatted at 2026-09-28 16:40:09 on dist-test-slave-nqgh"
server_key: "c44b8a0b7d7ff402a3701186209e80b4"
server_key_iv: "8b99cf3c9608b331748fd1f420bc4876"
server_key_version: "encryptionkey@0"
I20260928 16:40:09.168460 3696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "913581dad71b4327a1dba1cb8560b317"
format_stamp: "Formatted at 2026-09-28 16:40:09 on dist-test-slave-nqgh"
server_key: "c44b8a0b7d7ff402a3701186209e80b4"
server_key_iv: "8b99cf3c9608b331748fd1f420bc4876"
server_key_version: "encryptionkey@0"
I20260928 16:40:09.173897 3696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:40:09.177412 3711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:09.179405 3696 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:40:09.179612 3696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "913581dad71b4327a1dba1cb8560b317"
format_stamp: "Formatted at 2026-09-28 16:40:09 on dist-test-slave-nqgh"
server_key: "c44b8a0b7d7ff402a3701186209e80b4"
server_key_iv: "8b99cf3c9608b331748fd1f420bc4876"
server_key_version: "encryptionkey@0"
I20260928 16:40:09.179775 3696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:40:09.195835 3696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:40:09.200464 3696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:40:09.200781 3696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:40:09.210260 3696 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:41521
I20260928 16:40:09.210314 3763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:41521 every 8 connection(s)
I20260928 16:40:09.211642 3696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:40:09.215194 3764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:40:09.221510 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 3696
I20260928 16:40:09.221662 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:40:09.221707 3764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317: Bootstrap starting.
I20260928 16:40:09.221992 25885 external_mini_cluster.cc:1428] Setting key ee61a0215755de28895a3bac0ab4aa9e
I20260928 16:40:09.225064 3764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317: Neither blocks nor log segments found. Creating new log.
I20260928 16:40:09.225930 3764 log.cc:824] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317: Log is configured to *not* fsync() on all Append() calls
I20260928 16:40:09.228928 3764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317: No bootstrap required, opened a new log
Sep 28 16:40:09 dist-test-slave-nqgh krb5kdc[3683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:40:09.233777 3764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913581dad71b4327a1dba1cb8560b317" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41521 } }
I20260928 16:40:09.234154 3764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:40:09.234200 3764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913581dad71b4327a1dba1cb8560b317, State: Initialized, Role: FOLLOWER
I20260928 16:40:09.234768 3764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913581dad71b4327a1dba1cb8560b317" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41521 } }
I20260928 16:40:09.234951 3764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:40:09.235031 3764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:40:09.235144 3764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:40:09.236244 3764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913581dad71b4327a1dba1cb8560b317" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41521 } }
I20260928 16:40:09.236588 3764 leader_election.cc:302] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 913581dad71b4327a1dba1cb8560b317; no voters:
I20260928 16:40:09.236975 3764 leader_election.cc:288] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:40:09.237146 3769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:40:09.237471 3769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317 [term 1 LEADER]: Becoming Leader. State: Replica: 913581dad71b4327a1dba1cb8560b317, State: Running, Role: LEADER
I20260928 16:40:09.238036 3769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913581dad71b4327a1dba1cb8560b317" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41521 } }
I20260928 16:40:09.238655 3764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:40:09.240417 3770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "913581dad71b4327a1dba1cb8560b317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913581dad71b4327a1dba1cb8560b317" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41521 } } }
I20260928 16:40:09.240619 3770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317 [sys.catalog]: This master's current role is: LEADER
I20260928 16:40:09.240988 3771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 913581dad71b4327a1dba1cb8560b317. Latest consensus state: current_term: 1 leader_uuid: "913581dad71b4327a1dba1cb8560b317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913581dad71b4327a1dba1cb8560b317" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41521 } } }
I20260928 16:40:09.241091 3771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317 [sys.catalog]: This master's current role is: LEADER
I20260928 16:40:09.241441 3778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:40:09.241482 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:09.223898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55866 (local address 127.25.71.126:41521)
0928 16:40:09.224495 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:09.224516 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:09.224568 (+ 52us) server_negotiation.cc:407] Connection header received
0928 16:40:09.225512 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:09.225541 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:09.226049 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:09.226528 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:09.227438 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:09.228643 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:09.229982 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:09.230732 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:09.230897 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:09.234312 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:40:09.234323 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:40:09.234348 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:40:09.234403 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:40:09.236628 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:40:09.237324 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:40:09.237333 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:40:09.237341 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:40:09.237551 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:40:09.238045 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:40:09.238048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:40:09.238050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:40:09.238806 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:40:09.239061 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 16:40:09.239647 (+ 586us) server_negotiation.cc:299] Negotiation successful
0928 16:40:09.240093 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":182,"threads_started":1}
I20260928 16:40:09.246419 3778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:40:09.253923 3778 catalog_manager.cc:1347] Generated new cluster ID: 04bda913510b4b23b1fcafc91695ad5f
I20260928 16:40:09.254012 3778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:40:09.262425 3778 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:40:09.264173 3778 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:40:09.274227 3778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317: Generated new TSK 0
I20260928 16:40:09.275566 3778 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
I20260928 16:40:09.319113 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41521
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:40:09.447144 3790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:40:09.447537 3790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:40:09.447607 3790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:40:09.452147 3790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:40:09.452265 3790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:40:09.452358 3790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:40:09.459245 3790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:40:09.460536 3790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:40:09.461722 3790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:40:09.472929 3796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:40:09.472956 3795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:40:09.472956 3798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:40:09.473379 3790 server_base.cc:1034] running on GCE node
I20260928 16:40:09.473918 3790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:40:09.474787 3790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:40:09.476037 3790 hybrid_clock.cc:648] HybridClock initialized: now 1790613609476006 us; error 54 us; skew 500 ppm
Sep 28 16:40:09 dist-test-slave-nqgh krb5kdc[3683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613609, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:40:09.491132 3790 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:40:09.492260 3790 webserver.cc:460] Webserver started at http://127.25.71.65:43205/ using document root <none> and password file <none>
I20260928 16:40:09.492915 3790 fs_manager.cc:359] Metadata directory not provided
I20260928 16:40:09.492988 3790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:40:09.493199 3790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:40:09.495493 3790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "179b8cff71b4415ab2b2fae9affef9d5"
format_stamp: "Formatted at 2026-09-28 16:40:09 on dist-test-slave-nqgh"
server_key: "8fa45a82bf3ccd932d6da5602662442c"
server_key_iv: "7bfb80a047eb56222b7319af6fde28ee"
server_key_version: "encryptionkey@0"
I20260928 16:40:09.496107 3790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "179b8cff71b4415ab2b2fae9affef9d5"
format_stamp: "Formatted at 2026-09-28 16:40:09 on dist-test-slave-nqgh"
server_key: "8fa45a82bf3ccd932d6da5602662442c"
server_key_iv: "7bfb80a047eb56222b7319af6fde28ee"
server_key_version: "encryptionkey@0"
I20260928 16:40:09.500494 3790 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:40:09.503688 3805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:09.505342 3790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:40:09.505522 3790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "179b8cff71b4415ab2b2fae9affef9d5"
format_stamp: "Formatted at 2026-09-28 16:40:09 on dist-test-slave-nqgh"
server_key: "8fa45a82bf3ccd932d6da5602662442c"
server_key_iv: "7bfb80a047eb56222b7319af6fde28ee"
server_key_version: "encryptionkey@0"
I20260928 16:40:09.505650 3790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:40:09.522188 3790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:40:09.526540 3790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:40:09.526790 3790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:40:09.527454 3790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:40:09.528399 3790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:40:09.528450 3790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:09.528508 3790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:40:09.528538 3790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:09.538908 3918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:45543 every 8 connection(s)
I20260928 16:40:09.538915 3790 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:45543
I20260928 16:40:09.540144 3790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:40:09.549374 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 3790
I20260928 16:40:09.549546 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:40:09.549918 25885 external_mini_cluster.cc:1428] Setting key a58e70a89516e7b907478f4a0c486e06
Sep 28 16:40:09 dist-test-slave-nqgh krb5kdc[3683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613609, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:40:09.560962 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:09.542930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:39091 (local address 127.25.71.126:41521)
0928 16:40:09.543137 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:09.543141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:09.544182 (+ 1041us) server_negotiation.cc:407] Connection header received
0928 16:40:09.545969 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:09.545972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:09.546048 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:09.546130 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:09.547836 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:09.548444 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:09.550399 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:09.550890 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:09.551070 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:09.555193 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:40:09.555198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:40:09.555201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:40:09.555241 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:40:09.556903 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:40:09.557968 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:40:09.557972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:40:09.557974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:40:09.558200 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:40:09.558823 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:40:09.558827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:40:09.558828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:40:09.559106 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:40:09.559201 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 16:40:09.560468 (+ 1267us) server_negotiation.cc:299] Negotiation successful
0928 16:40:09.560738 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 16:40:09.562165 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:09.543294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:41521 (local address 127.25.71.65:39091)
0928 16:40:09.543946 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:09.544014 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:09.545669 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:09.546399 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:09.546409 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:09.546953 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:09.547631 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:09.547650 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:09.548594 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:09.548597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:09.550072 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:09.550088 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:09.551484 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:09.551490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:09.551736 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:09.552494 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:40:09.552518 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:09.554829 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:40:09.557107 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:40:09.557119 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:40:09.557148 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:40:09.557739 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:40:09.558344 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:40:09.558348 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:40:09.558350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:40:09.558683 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:40:09.559406 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:40:09.559419 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:40:09.560044 (+ 625us) client_negotiation.cc:769] Sending connection context
0928 16:40:09.560353 (+ 309us) client_negotiation.cc:240] Negotiation successful
0928 16:40:09.560887 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":152,"threads_started":1}
WARNING: no policy specified for kudu/127.25.71.66@KRBTEST.COM; defaulting to no policy
I20260928 16:40:09.564421 3919 heartbeater.cc:347] Connected to a master server at 127.25.71.126:41521
I20260928 16:40:09.564739 3919 heartbeater.cc:464] Registering TS with master...
I20260928 16:40:09.565409 3919 heartbeater.cc:511] Master 127.25.71.126:41521 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.66@KRBTEST.COM" created.
I20260928 16:40:09.567261 3728 ts_manager.cc:194] Registered new tserver with Master: 179b8cff71b4415ab2b2fae9affef9d5 (127.25.71.65:45543)
I20260928 16:40:09.568853 3728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:39091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
I20260928 16:40:09.588456 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41521
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:40:09.726842 3924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:40:09.727239 3924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:40:09.727293 3924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:40:09.731802 3924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:40:09.731905 3924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:40:09.731976 3924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:40:09.738286 3924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:40:09.739885 3924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:40:09.741374 3924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:40:09.753571 3932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:40:09.753594 3930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:40:09.753614 3929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:40:09.753933 3924 server_base.cc:1034] running on GCE node
I20260928 16:40:09.754566 3924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:40:09.755537 3924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:40:09.756760 3924 hybrid_clock.cc:648] HybridClock initialized: now 1790613609756723 us; error 50 us; skew 500 ppm
Sep 28 16:40:09 dist-test-slave-nqgh krb5kdc[3683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613609, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:40:09.760473 3924 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:40:09.761839 3924 webserver.cc:460] Webserver started at http://127.25.71.66:34035/ using document root <none> and password file <none>
I20260928 16:40:09.762562 3924 fs_manager.cc:359] Metadata directory not provided
I20260928 16:40:09.762682 3924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:40:09.762984 3924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:40:09.765787 3924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "5feb502d2b9149d0981c256f37b28375"
format_stamp: "Formatted at 2026-09-28 16:40:09 on dist-test-slave-nqgh"
server_key: "b07ed8f015cf9fa2f6c63bcee9fa11c9"
server_key_iv: "3b92ec03d70e24d279890efbc8561be2"
server_key_version: "encryptionkey@0"
I20260928 16:40:09.766736 3924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "5feb502d2b9149d0981c256f37b28375"
format_stamp: "Formatted at 2026-09-28 16:40:09 on dist-test-slave-nqgh"
server_key: "b07ed8f015cf9fa2f6c63bcee9fa11c9"
server_key_iv: "3b92ec03d70e24d279890efbc8561be2"
server_key_version: "encryptionkey@0"
I20260928 16:40:09.772670 3924 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 16:40:09.776605 3939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:09.778400 3924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:40:09.778573 3924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "5feb502d2b9149d0981c256f37b28375"
format_stamp: "Formatted at 2026-09-28 16:40:09 on dist-test-slave-nqgh"
server_key: "b07ed8f015cf9fa2f6c63bcee9fa11c9"
server_key_iv: "3b92ec03d70e24d279890efbc8561be2"
server_key_version: "encryptionkey@0"
I20260928 16:40:09.778771 3924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:40:09.794025 3924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:40:09.797466 3924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:40:09.797664 3924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:40:09.798316 3924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:40:09.799309 3924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:40:09.799363 3924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:09.799431 3924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:40:09.799458 3924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:09.809516 3924 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:33293
I20260928 16:40:09.809536 4052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:33293 every 8 connection(s)
I20260928 16:40:09.810645 3924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:40:09.817320 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 3924
I20260928 16:40:09.817464 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:40:09.817816 25885 external_mini_cluster.cc:1428] Setting key 9a54f2da3fe5b588dcec11e4c3d03be3
Sep 28 16:40:09 dist-test-slave-nqgh krb5kdc[3683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613609, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:40:09.829056 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:09.813238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:59717 (local address 127.25.71.126:41521)
0928 16:40:09.813475 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:09.813481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:09.814699 (+ 1218us) server_negotiation.cc:407] Connection header received
0928 16:40:09.816435 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:09.816439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:09.816519 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:09.816628 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:09.818215 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:09.818895 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:09.820398 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:09.820762 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:09.820914 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:09.824273 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:40:09.824279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:40:09.824282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:40:09.824329 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:40:09.826194 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:40:09.826981 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:40:09.826986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:40:09.826988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:40:09.827130 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:40:09.827582 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:40:09.827585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:40:09.827587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:40:09.827895 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:40:09.828009 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 16:40:09.828655 (+ 646us) server_negotiation.cc:299] Negotiation successful
0928 16:40:09.828827 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 16:40:09.829855 4055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:09.813744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:41521 (local address 127.25.71.66:59717)
0928 16:40:09.814468 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:09.814536 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:09.816136 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:09.816900 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:09.816909 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:09.817383 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:09.818012 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:09.818031 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:09.819060 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:09.819064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:09.820206 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:09.820218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:09.820992 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:09.820996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:09.821176 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:09.821985 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:40:09.822005 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:09.823964 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:40:09.826378 (+ 2414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:40:09.826384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:40:09.826402 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:40:09.826833 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:40:09.827257 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:40:09.827259 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:40:09.827260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:40:09.827455 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:40:09.828029 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:40:09.828036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:40:09.828357 (+ 321us) client_negotiation.cc:769] Sending connection context
0928 16:40:09.828538 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 16:40:09.828863 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":185,"threads_started":1}
WARNING: no policy specified for kudu/127.25.71.67@KRBTEST.COM; defaulting to no policy
I20260928 16:40:09.831534 4053 heartbeater.cc:347] Connected to a master server at 127.25.71.126:41521
I20260928 16:40:09.831961 4053 heartbeater.cc:464] Registering TS with master...
I20260928 16:40:09.832733 4053 heartbeater.cc:511] Master 127.25.71.126:41521 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.67@KRBTEST.COM" created.
I20260928 16:40:09.834323 3728 ts_manager.cc:194] Registered new tserver with Master: 5feb502d2b9149d0981c256f37b28375 (127.25.71.66:33293)
I20260928 16:40:09.835100 3728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:59717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
I20260928 16:40:09.855743 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41521
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:40:10.006569 4058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:40:10.006999 4058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:40:10.007051 4058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:40:10.011742 4058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:40:10.011855 4058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:40:10.011927 4058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:40:10.018208 4058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:40:10.019798 4058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:40:10.021201 4058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:40:10.034042 4066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:40:10.034092 4063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:40:10.034092 4064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:40:10.034397 4058 server_base.cc:1034] running on GCE node
I20260928 16:40:10.035115 4058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:40:10.036067 4058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:40:10.037294 4058 hybrid_clock.cc:648] HybridClock initialized: now 1790613610037273 us; error 50 us; skew 500 ppm
Sep 28 16:40:10 dist-test-slave-nqgh krb5kdc[3683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613610, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:40:10.042023 4058 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:40:10.043525 4058 webserver.cc:460] Webserver started at http://127.25.71.67:36871/ using document root <none> and password file <none>
I20260928 16:40:10.044320 4058 fs_manager.cc:359] Metadata directory not provided
I20260928 16:40:10.044410 4058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:40:10.044672 4058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:40:10.047863 4058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "fbd8f1808ca24dce910a94559755345c"
format_stamp: "Formatted at 2026-09-28 16:40:10 on dist-test-slave-nqgh"
server_key: "f843c3919a10a33222be8adafb59979e"
server_key_iv: "9b4b532e5107ec47d868fbb0a66a048f"
server_key_version: "encryptionkey@0"
I20260928 16:40:10.048847 4058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "fbd8f1808ca24dce910a94559755345c"
format_stamp: "Formatted at 2026-09-28 16:40:10 on dist-test-slave-nqgh"
server_key: "f843c3919a10a33222be8adafb59979e"
server_key_iv: "9b4b532e5107ec47d868fbb0a66a048f"
server_key_version: "encryptionkey@0"
I20260928 16:40:10.055274 4058 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 16:40:10.059573 4073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:10.061872 4058 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:40:10.062129 4058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "fbd8f1808ca24dce910a94559755345c"
format_stamp: "Formatted at 2026-09-28 16:40:10 on dist-test-slave-nqgh"
server_key: "f843c3919a10a33222be8adafb59979e"
server_key_iv: "9b4b532e5107ec47d868fbb0a66a048f"
server_key_version: "encryptionkey@0"
I20260928 16:40:10.062318 4058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:40:10.125571 4058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:40:10.129860 4058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:40:10.130127 4058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:40:10.130980 4058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:40:10.132102 4058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:40:10.132182 4058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:10.132249 4058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:40:10.132274 4058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:40:10.145695 4058 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:46841
I20260928 16:40:10.145769 4186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:46841 every 8 connection(s)
I20260928 16:40:10.147083 4058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:40:10.156230 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 4058
I20260928 16:40:10.156359 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:40:10.156672 25885 external_mini_cluster.cc:1428] Setting key d269e9bbb03a89180894a0f0d173bdb4
Sep 28 16:40:10 dist-test-slave-nqgh krb5kdc[3683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613610, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:40:10.165853 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:10.149563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:37813 (local address 127.25.71.126:41521)
0928 16:40:10.149776 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:10.149780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:10.150989 (+ 1209us) server_negotiation.cc:407] Connection header received
0928 16:40:10.152840 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:10.152844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:10.152917 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:10.153022 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:10.154870 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.155506 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.157146 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.157487 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.157588 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:10.161045 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:40:10.161048 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:40:10.161051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:40:10.161087 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:40:10.162994 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:40:10.163857 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:40:10.163861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:40:10.163862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:40:10.163964 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:40:10.164453 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:40:10.164455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:40:10.164456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:40:10.164662 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:40:10.164742 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 16:40:10.165558 (+ 816us) server_negotiation.cc:299] Negotiation successful
0928 16:40:10.165700 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:40:10.167094 4189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:10.149920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:41521 (local address 127.25.71.67:37813)
0928 16:40:10.150729 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:10.150817 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:10.152515 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:10.153334 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:10.153343 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:10.153874 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:40:10.154583 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:10.154604 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.155696 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.155700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:10.156972 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:10.156986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.157716 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.157719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:10.157969 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:10.158765 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:40:10.158786 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:40:10.160787 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:40:10.163224 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:40:10.163234 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:40:10.163257 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:40:10.163680 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:40:10.164079 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:40:10.164082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:40:10.164084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:40:10.164345 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:40:10.164754 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:40:10.164762 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:40:10.165132 (+ 370us) client_negotiation.cc:769] Sending connection context
0928 16:40:10.165428 (+ 296us) client_negotiation.cc:240] Negotiation successful
0928 16:40:10.165830 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":168,"threads_started":1}
I20260928 16:40:10.168947 4187 heartbeater.cc:347] Connected to a master server at 127.25.71.126:41521
I20260928 16:40:10.169299 4187 heartbeater.cc:464] Registering TS with master...
I20260928 16:40:10.170040 4187 heartbeater.cc:511] Master 127.25.71.126:41521 requested a full tablet report, sending...
I20260928 16:40:10.171633 3728 ts_manager.cc:194] Registered new tserver with Master: fbd8f1808ca24dce910a94559755345c (127.25.71.67:46841)
I20260928 16:40:10.172384 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:40:10.172472 3728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:37813
I20260928 16:40:10.185447 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:10.175005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55882 (local address 127.25.71.126:41521)
0928 16:40:10.175187 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:10.175193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:10.175628 (+ 435us) server_negotiation.cc:407] Connection header received
0928 16:40:10.175749 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:10.175752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:10.175807 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:10.175896 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:40:10.176766 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.177417 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.178607 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.178930 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.179126 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:10.180797 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:40:10.180801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:40:10.180804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:40:10.180843 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:40:10.182843 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:40:10.183526 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:40:10.183529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:40:10.183532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:40:10.183652 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:40:10.184206 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:40:10.184210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:40:10.184213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:40:10.184573 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:40:10.184727 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 16:40:10.185091 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 16:40:10.185260 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:40:10.189141 3728 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:40:10.193277 3728 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:40:10.213608 4201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:10.206380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:45543 (local address 127.0.0.1:54908)
0928 16:40:10.207450 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:10.207468 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:10.207619 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:10.208137 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:10.208143 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:10.208169 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:40:10.208445 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:10.208454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.210018 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.210022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:10.211505 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:10.211516 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.212936 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.212940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:10.213079 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:10.213143 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 16:40:10.213239 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 16:40:10.213407 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"mutex_wait_us":742,"thread_start_us":129,"threads_started":1}
I20260928 16:40:10.214696 4200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:10.206722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54908 (local address 127.25.71.65:45543)
0928 16:40:10.207400 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:10.207409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:10.207589 (+ 180us) server_negotiation.cc:407] Connection header received
0928 16:40:10.207762 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:10.207766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:10.207968 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:10.208118 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:40:10.208587 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.209890 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.211634 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.212757 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.214126 (+ 1369us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:10.214323 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 16:40:10.214386 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 16:40:10.214508 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":148,"threads_started":1}
I20260928 16:40:10.215045 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:10.205937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:46841 (local address 127.0.0.1:44352)
0928 16:40:10.206684 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:10.206764 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:10.206937 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:10.208225 (+ 1288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:10.208228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:10.208243 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:40:10.208449 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:10.208457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.211633 (+ 3176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.211636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:10.213026 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:10.213036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.214475 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.214479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:10.214575 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:10.214600 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 16:40:10.214698 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 16:40:10.214782 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":6,"thread_start_us":221,"threads_started":1}
I20260928 16:40:10.215726 4202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:10.207087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:33293 (local address 127.0.0.1:46754)
0928 16:40:10.208075 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:10.208089 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:10.208194 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:10.208877 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:10.208880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:10.208903 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:40:10.209009 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:10.209016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.210791 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.210795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:10.212169 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:10.212182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.215236 (+ 3054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.215240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:10.215369 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:10.215397 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 16:40:10.215458 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 16:40:10.215556 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":921,"thread_start_us":121,"threads_started":1}
I20260928 16:40:10.216185 4199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:10.206369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.25.71.67:46841)
0928 16:40:10.207606 (+ 1237us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:10.207620 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:10.207651 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:40:10.207757 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:10.207763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:10.208028 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:10.208699 (+ 671us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:40:10.208938 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.211464 (+ 2526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.213205 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.214310 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.215468 (+ 1158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:10.215630 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 16:40:10.215688 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 16:40:10.215796 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1106,"thread_start_us":205,"threads_started":1}
I20260928 16:40:10.216565 4203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:10.207381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46754 (local address 127.25.71.66:33293)
0928 16:40:10.207930 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:10.207941 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:10.208284 (+ 343us) server_negotiation.cc:407] Connection header received
0928 16:40:10.208369 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:10.208374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:10.208651 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:10.208873 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:40:10.209400 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.210595 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.212364 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.214918 (+ 2554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.215040 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:10.216112 (+ 1072us) server_negotiation.cc:1035] Waiting for connection context
0928 16:40:10.216179 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 16:40:10.216391 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":162,"threads_started":1}
I20260928 16:40:10.218531 3853 tablet_service.cc:1467] Processing CreateTablet for tablet f3bded419b37434293efad73abf219aa (DEFAULT_TABLE table=test-table [id=a3fd0b90f4cb41aca11451daac6b0aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:40:10.220081 3853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3bded419b37434293efad73abf219aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:40:10.220108 4121 tablet_service.cc:1467] Processing CreateTablet for tablet f3bded419b37434293efad73abf219aa (DEFAULT_TABLE table=test-table [id=a3fd0b90f4cb41aca11451daac6b0aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:40:10.220649 3987 tablet_service.cc:1467] Processing CreateTablet for tablet f3bded419b37434293efad73abf219aa (DEFAULT_TABLE table=test-table [id=a3fd0b90f4cb41aca11451daac6b0aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:40:10.221566 4121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3bded419b37434293efad73abf219aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:40:10.221801 3987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3bded419b37434293efad73abf219aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:40:10.229137 4204 tablet_bootstrap.cc:492] T f3bded419b37434293efad73abf219aa P 179b8cff71b4415ab2b2fae9affef9d5: Bootstrap starting.
I20260928 16:40:10.232182 4206 tablet_bootstrap.cc:492] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c: Bootstrap starting.
I20260928 16:40:10.233124 4204 tablet_bootstrap.cc:654] T f3bded419b37434293efad73abf219aa P 179b8cff71b4415ab2b2fae9affef9d5: Neither blocks nor log segments found. Creating new log.
I20260928 16:40:10.234201 4204 log.cc:824] T f3bded419b37434293efad73abf219aa P 179b8cff71b4415ab2b2fae9affef9d5: Log is configured to *not* fsync() on all Append() calls
I20260928 16:40:10.234903 4205 tablet_bootstrap.cc:492] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375: Bootstrap starting.
I20260928 16:40:10.235069 4206 tablet_bootstrap.cc:654] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c: Neither blocks nor log segments found. Creating new log.
I20260928 16:40:10.235980 4206 log.cc:824] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:40:10.237438 4204 tablet_bootstrap.cc:492] T f3bded419b37434293efad73abf219aa P 179b8cff71b4415ab2b2fae9affef9d5: No bootstrap required, opened a new log
I20260928 16:40:10.237738 4204 ts_tablet_manager.cc:1397] T f3bded419b37434293efad73abf219aa P 179b8cff71b4415ab2b2fae9affef9d5: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260928 16:40:10.238204 4206 tablet_bootstrap.cc:492] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c: No bootstrap required, opened a new log
I20260928 16:40:10.238461 4206 ts_tablet_manager.cc:1397] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:40:10.238713 4205 tablet_bootstrap.cc:654] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375: Neither blocks nor log segments found. Creating new log.
I20260928 16:40:10.240130 4205 log.cc:824] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375: Log is configured to *not* fsync() on all Append() calls
I20260928 16:40:10.242496 4206 raft_consensus.cc:348] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd8f1808ca24dce910a94559755345c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 46841 } } peers { permanent_uuid: "5feb502d2b9149d0981c256f37b28375" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33293 } } peers { permanent_uuid: "179b8cff71b4415ab2b2fae9affef9d5" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 45543 } }
I20260928 16:40:10.242865 4206 raft_consensus.cc:374] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:40:10.242928 4206 raft_consensus.cc:729] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd8f1808ca24dce910a94559755345c, State: Initialized, Role: FOLLOWER
I20260928 16:40:10.243439 4206 consensus_queue.cc:260] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd8f1808ca24dce910a94559755345c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 46841 } } peers { permanent_uuid: "5feb502d2b9149d0981c256f37b28375" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33293 } } peers { permanent_uuid: "179b8cff71b4415ab2b2fae9affef9d5" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 45543 } }
I20260928 16:40:10.243230 4204 raft_consensus.cc:348] T f3bded419b37434293efad73abf219aa P 179b8cff71b4415ab2b2fae9affef9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd8f1808ca24dce910a94559755345c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 46841 } } peers { permanent_uuid: "5feb502d2b9149d0981c256f37b28375" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33293 } } peers { permanent_uuid: "179b8cff71b4415ab2b2fae9affef9d5" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 45543 } }
I20260928 16:40:10.243599 4204 raft_consensus.cc:374] T f3bded419b37434293efad73abf219aa P 179b8cff71b4415ab2b2fae9affef9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:40:10.243628 4205 tablet_bootstrap.cc:492] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375: No bootstrap required, opened a new log
I20260928 16:40:10.243690 4204 raft_consensus.cc:729] T f3bded419b37434293efad73abf219aa P 179b8cff71b4415ab2b2fae9affef9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179b8cff71b4415ab2b2fae9affef9d5, State: Initialized, Role: FOLLOWER
I20260928 16:40:10.243913 4205 ts_tablet_manager.cc:1397] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260928 16:40:10.244277 4204 consensus_queue.cc:260] T f3bded419b37434293efad73abf219aa P 179b8cff71b4415ab2b2fae9affef9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd8f1808ca24dce910a94559755345c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 46841 } } peers { permanent_uuid: "5feb502d2b9149d0981c256f37b28375" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33293 } } peers { permanent_uuid: "179b8cff71b4415ab2b2fae9affef9d5" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 45543 } }
I20260928 16:40:10.244606 4206 ts_tablet_manager.cc:1428] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 16:40:10.245042 4187 heartbeater.cc:503] Master 127.25.71.126:41521 was elected leader, sending a full tablet report...
I20260928 16:40:10.245190 3919 heartbeater.cc:503] Master 127.25.71.126:41521 was elected leader, sending a full tablet report...
I20260928 16:40:10.245687 4204 ts_tablet_manager.cc:1428] T f3bded419b37434293efad73abf219aa P 179b8cff71b4415ab2b2fae9affef9d5: Time spent starting tablet: real 0.008s user 0.007s sys 0.002s
I20260928 16:40:10.249168 4205 raft_consensus.cc:348] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd8f1808ca24dce910a94559755345c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 46841 } } peers { permanent_uuid: "5feb502d2b9149d0981c256f37b28375" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33293 } } peers { permanent_uuid: "179b8cff71b4415ab2b2fae9affef9d5" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 45543 } }
I20260928 16:40:10.249634 4205 raft_consensus.cc:374] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:40:10.249720 4205 raft_consensus.cc:729] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5feb502d2b9149d0981c256f37b28375, State: Initialized, Role: FOLLOWER
I20260928 16:40:10.250463 4205 consensus_queue.cc:260] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd8f1808ca24dce910a94559755345c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 46841 } } peers { permanent_uuid: "5feb502d2b9149d0981c256f37b28375" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33293 } } peers { permanent_uuid: "179b8cff71b4415ab2b2fae9affef9d5" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 45543 } }
I20260928 16:40:10.252126 4205 ts_tablet_manager.cc:1428] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260928 16:40:10.252628 4053 heartbeater.cc:503] Master 127.25.71.126:41521 was elected leader, sending a full tablet report...
W20260928 16:40:10.292050 3920 tablet.cc:2369] T f3bded419b37434293efad73abf219aa P 179b8cff71b4415ab2b2fae9affef9d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:40:10.312309 4054 tablet.cc:2369] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:40:10.398463 4188 tablet.cc:2369] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:40:10.427711 4210 raft_consensus.cc:482] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:40:10.427989 4210 raft_consensus.cc:504] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd8f1808ca24dce910a94559755345c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 46841 } } peers { permanent_uuid: "5feb502d2b9149d0981c256f37b28375" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33293 } } peers { permanent_uuid: "179b8cff71b4415ab2b2fae9affef9d5" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 45543 } }
I20260928 16:40:10.429720 4210 leader_election.cc:288] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5feb502d2b9149d0981c256f37b28375 (127.25.71.66:33293), 179b8cff71b4415ab2b2fae9affef9d5 (127.25.71.65:45543)
I20260928 16:40:10.436195 4203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:10.430486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:43669 (local address 127.25.71.66:33293)
0928 16:40:10.430717 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:10.430721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:10.431683 (+ 962us) server_negotiation.cc:407] Connection header received
0928 16:40:10.431782 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:10.431785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:10.431855 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:10.431965 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:40:10.432311 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.432941 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.434538 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.435297 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.435487 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:10.435561 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 16:40:10.435913 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 16:40:10.436022 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 16:40:10.436233 4200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:10.430522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:46075 (local address 127.25.71.65:45543)
0928 16:40:10.430724 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:10.430730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:10.430753 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:40:10.430889 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:10.430893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:10.430958 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:10.431043 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:40:10.431692 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.432596 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.434538 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.435305 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.435487 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:10.435561 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 16:40:10.435906 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 16:40:10.436115 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:40:10.436316 4189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:10.430361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:45543 (local address 127.25.71.67:46075)
0928 16:40:10.430524 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:10.430545 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:10.430761 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:10.431131 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:10.431137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:10.431184 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:40:10.431340 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:10.431350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.432778 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.432781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:10.434312 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:10.434327 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.435539 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.435544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:10.435704 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:10.435732 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 16:40:10.435841 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 16:40:10.435980 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":119}
I20260928 16:40:10.436334 4213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:10.430361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:33293 (local address 127.25.71.67:43669)
0928 16:40:10.431456 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:10.431474 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:40:10.431576 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:40:10.432022 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:40:10.432024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:40:10.432044 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:40:10.432165 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:10.432170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.433079 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.433083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:10.434312 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:40:10.434327 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.435539 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.435544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:40:10.435698 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:10.435734 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 16:40:10.435841 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 16:40:10.435980 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"mutex_wait_us":505,"thread_start_us":200,"threads_started":1}
I20260928 16:40:10.437345 4007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3bded419b37434293efad73abf219aa" candidate_uuid: "fbd8f1808ca24dce910a94559755345c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5feb502d2b9149d0981c256f37b28375" is_pre_election: true
I20260928 16:40:10.437345 3873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3bded419b37434293efad73abf219aa" candidate_uuid: "fbd8f1808ca24dce910a94559755345c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "179b8cff71b4415ab2b2fae9affef9d5" is_pre_election: true
I20260928 16:40:10.437798 3873 raft_consensus.cc:2454] T f3bded419b37434293efad73abf219aa P 179b8cff71b4415ab2b2fae9affef9d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbd8f1808ca24dce910a94559755345c in term 0.
I20260928 16:40:10.437798 4007 raft_consensus.cc:2454] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbd8f1808ca24dce910a94559755345c in term 0.
I20260928 16:40:10.438534 4076 leader_election.cc:302] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5feb502d2b9149d0981c256f37b28375, fbd8f1808ca24dce910a94559755345c; no voters:
I20260928 16:40:10.438979 4210 raft_consensus.cc:2781] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:40:10.439092 4210 raft_consensus.cc:482] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:40:10.439136 4210 raft_consensus.cc:3037] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:40:10.440481 4210 raft_consensus.cc:504] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd8f1808ca24dce910a94559755345c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 46841 } } peers { permanent_uuid: "5feb502d2b9149d0981c256f37b28375" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33293 } } peers { permanent_uuid: "179b8cff71b4415ab2b2fae9affef9d5" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 45543 } }
I20260928 16:40:10.440923 4210 leader_election.cc:288] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c [CANDIDATE]: Term 1 election: Requested vote from peers 5feb502d2b9149d0981c256f37b28375 (127.25.71.66:33293), 179b8cff71b4415ab2b2fae9affef9d5 (127.25.71.65:45543)
I20260928 16:40:10.441571 4007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3bded419b37434293efad73abf219aa" candidate_uuid: "fbd8f1808ca24dce910a94559755345c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5feb502d2b9149d0981c256f37b28375"
I20260928 16:40:10.441813 3873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3bded419b37434293efad73abf219aa" candidate_uuid: "fbd8f1808ca24dce910a94559755345c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "179b8cff71b4415ab2b2fae9affef9d5"
I20260928 16:40:10.441893 4007 raft_consensus.cc:3037] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:40:10.442000 3873 raft_consensus.cc:3037] T f3bded419b37434293efad73abf219aa P 179b8cff71b4415ab2b2fae9affef9d5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:40:10.443513 3873 raft_consensus.cc:2454] T f3bded419b37434293efad73abf219aa P 179b8cff71b4415ab2b2fae9affef9d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbd8f1808ca24dce910a94559755345c in term 1.
I20260928 16:40:10.443639 4007 raft_consensus.cc:2454] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbd8f1808ca24dce910a94559755345c in term 1.
I20260928 16:40:10.444340 4076 leader_election.cc:302] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5feb502d2b9149d0981c256f37b28375, fbd8f1808ca24dce910a94559755345c; no voters:
I20260928 16:40:10.444720 4210 raft_consensus.cc:2781] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:40:10.445057 4210 raft_consensus.cc:686] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c [term 1 LEADER]: Becoming Leader. State: Replica: fbd8f1808ca24dce910a94559755345c, State: Running, Role: LEADER
I20260928 16:40:10.445592 4210 consensus_queue.cc:237] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd8f1808ca24dce910a94559755345c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 46841 } } peers { permanent_uuid: "5feb502d2b9149d0981c256f37b28375" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33293 } } peers { permanent_uuid: "179b8cff71b4415ab2b2fae9affef9d5" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 45543 } }
I20260928 16:40:10.450222 3722 catalog_manager.cc:5495] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c reported cstate change: term changed from 0 to 1, leader changed from <none> to fbd8f1808ca24dce910a94559755345c (127.25.71.67). New cstate: current_term: 1 leader_uuid: "fbd8f1808ca24dce910a94559755345c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd8f1808ca24dce910a94559755345c" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5feb502d2b9149d0981c256f37b28375" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "179b8cff71b4415ab2b2fae9affef9d5" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 45543 } health_report { overall_health: UNKNOWN } } }
I20260928 16:40:10.469476 4199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:10.465009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.25.71.67:46841)
0928 16:40:10.465145 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 16:40:10.465149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:40:10.465166 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:40:10.465263 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:40:10.465265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:40:10.465334 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:40:10.465485 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:40:10.465830 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.466473 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.467613 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:40:10.467951 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:40:10.468051 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:40:10.468473 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:40:10.469012 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:40:10.469124 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:40:10.469244 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 16:40:10.469309 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:40:10.477799 4007 raft_consensus.cc:1261] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375 [term 1 FOLLOWER]: Refusing update from remote peer fbd8f1808ca24dce910a94559755345c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:40:10.477799 3873 raft_consensus.cc:1261] T f3bded419b37434293efad73abf219aa P 179b8cff71b4415ab2b2fae9affef9d5 [term 1 FOLLOWER]: Refusing update from remote peer fbd8f1808ca24dce910a94559755345c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:40:10.479054 4214 consensus_queue.cc:1035] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5feb502d2b9149d0981c256f37b28375" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:40:10.479348 4210 consensus_queue.cc:1035] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c [LEADER]: Connected to new peer: Peer: permanent_uuid: "179b8cff71b4415ab2b2fae9affef9d5" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 45543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:40:10.501292 3722 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:40:10.501544 3722 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:40:10.505151 3722 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 913581dad71b4327a1dba1cb8560b317: Sending DeleteTablet for 3 replicas of tablet f3bded419b37434293efad73abf219aa
I20260928 16:40:10.506428 3853 tablet_service.cc:1514] Processing DeleteTablet for tablet f3bded419b37434293efad73abf219aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:40:10 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:54908
I20260928 16:40:10.506448 3987 tablet_service.cc:1514] Processing DeleteTablet for tablet f3bded419b37434293efad73abf219aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:40:10 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:46754
I20260928 16:40:10.507385 4121 tablet_service.cc:1514] Processing DeleteTablet for tablet f3bded419b37434293efad73abf219aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:40:10 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:44352
I20260928 16:40:10.507537 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 3790
I20260928 16:40:10.508541 4229 tablet_replica.cc:330] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c: stopping tablet replica
I20260928 16:40:10.508991 4229 raft_consensus.cc:2229] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:40:10.509518 4229 raft_consensus.cc:2258] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:40:10.509613 4227 tablet_replica.cc:330] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375: stopping tablet replica
I20260928 16:40:10.510053 4227 raft_consensus.cc:2229] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:40:10.510486 4227 raft_consensus.cc:2258] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:40:10.511175 4229 ts_tablet_manager.cc:1905] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:40:10.512391 4227 ts_tablet_manager.cc:1905] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:40:10.514357 4229 ts_tablet_manager.cc:1918] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:40:10.514497 4229 log.cc:1196] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/f3bded419b37434293efad73abf219aa
I20260928 16:40:10.514896 4229 ts_tablet_manager.cc:1939] T f3bded419b37434293efad73abf219aa P fbd8f1808ca24dce910a94559755345c: Deleting consensus metadata
I20260928 16:40:10.515987 4227 ts_tablet_manager.cc:1918] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:40:10.516146 4227 log.cc:1196] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/f3bded419b37434293efad73abf219aa
I20260928 16:40:10.516527 4227 ts_tablet_manager.cc:1939] T f3bded419b37434293efad73abf219aa P 5feb502d2b9149d0981c256f37b28375: Deleting consensus metadata
I20260928 16:40:10.516479 3714 catalog_manager.cc:4841] TS fbd8f1808ca24dce910a94559755345c (127.25.71.67:46841): tablet f3bded419b37434293efad73abf219aa (table test-table [id=a3fd0b90f4cb41aca11451daac6b0aad]) successfully deleted
I20260928 16:40:10.518410 3714 catalog_manager.cc:4841] TS 5feb502d2b9149d0981c256f37b28375 (127.25.71.66:33293): tablet f3bded419b37434293efad73abf219aa (table test-table [id=a3fd0b90f4cb41aca11451daac6b0aad]) successfully deleted
W20260928 16:40:10.519102 3713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.71.65:45543 (error 108)
I20260928 16:40:10.519245 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 3924
I20260928 16:40:10.521497 4202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:40:10.521175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:45543 (local address 127.0.0.1:54912)
0928 16:40:10.521313 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 16:40:10.521393 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.65:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 16:40:10.521814 3713 catalog_manager.cc:4584] TS 179b8cff71b4415ab2b2fae9affef9d5 (127.25.71.65:45543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3bded419b37434293efad73abf219aa: Network error: Client connection negotiation failed: client connection to 127.25.71.65:45543: connect: Connection refused (error 111)
I20260928 16:40:10.527808 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 4058
I20260928 16:40:10.535434 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 3696
2026-09-28T16:40:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2674 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:40:10 dist-test-slave-nqgh krb5kdc[4233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:40:10 dist-test-slave-nqgh krb5kdc[4233](info): set up 2 sockets
Sep 28 16:40:10 dist-test-slave-nqgh krb5kdc[4233](info): commencing operation
krb5kdc: starting...
W20260928 16:40:11.521250 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:40:11 dist-test-slave-nqgh krb5kdc[4233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:40:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:40:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.25.71.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.25.71.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.25.71.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/rangeradmin_127.25.71.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.25.71.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/rangeradmin_127.25.71.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.25.71.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.25.71.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.25.71.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/rangerlookup_127.25.71.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.25.71.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/rangerlookup_127.25.71.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.71.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.71.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.71.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/HTTP_127.25.71.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.71.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/HTTP_127.25.71.84@KRBTEST.COM.keytab.
I20260928 16:40:11.727360 25885 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskp76gbp/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/postgres -l logfile start
2026-09-28 16:40:14.951 UTC [4263] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 16:40:14.951 UTC [4263] LOG: listening on IPv4 address "127.25.71.84", port 43533
2026-09-28 16:40:14.955 UTC [4263] LOG: listening on Unix socket "/tmp/.s.PGSQL.43533"
2026-09-28 16:40:14.977 UTC [4266] LOG: database system was shut down at 2026-09-28 16:40:12 UTC
2026-09-28 16:40:14.982 UTC [4263] LOG: database system is ready to accept connections
I20260928 16:40:15.857403 25885 mini_postgres.cc:96] Postgres bound to 43533
2026-09-28 16:40:15.866 UTC [4274] FATAL: database "slave" does not exist
127.25.71.84:43533 - accepting connections
I20260928 16:40:15.867439 25885 mini_ranger.cc:161] Starting Ranger...
I20260928 16:40:15.887128 25885 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 16:40:16.568481 25885 mini_ranger.cc:87] Created ranger Postgres database
I20260928 16:40:16.568622 25885 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ranger-admin
2026-09-28 16:40:17,106 [I] DB FLAVOR :POSTGRES
2026-09-28 16:40:17,106 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 16:40:17,106 [I] Checking connection..
2026-09-28 16:40:17,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 16:40:17,444 [I] Checking connection passed.
2026-09-28 16:40:17,444 [I] --------- Verifying version history table ---------
2026-09-28 16:40:17,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:40:17,752 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 16:40:17,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:40:18,045 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 16:40:18,045 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 16:40:18,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 16:40:18.357 UTC [4338] WARNING: there is no transaction in progress
2026-09-28 16:40:18,366 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 16:40:18,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:40:18,662 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 16:40:18,663 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 16:40:18,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 16:40:18,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 16:40:19,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nqgh.c.gcp-upstream.internal','N') ;"
2026-09-28 16:40:19,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 16:40:19,798 [I] Table x_portal_user does not exist in database ranger
2026-09-28 16:40:19,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 16:40:20,099 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 16:40:20,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 16:40:20,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 16:40:20,610 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 16:40:20,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 16:40:21.891 UTC [4446] WARNING: there is no transaction in progress
2026-09-28 16:40:21.913 UTC [4446] WARNING: there is no transaction in progress
2026-09-28 16:40:21.934 UTC [4446] WARNING: there is no transaction in progress
2026-09-28 16:40:21.953 UTC [4446] WARNING: there is no transaction in progress
2026-09-28 16:40:21.977 UTC [4446] WARNING: there is no transaction in progress
2026-09-28 16:40:22.000 UTC [4446] WARNING: there is no transaction in progress
2026-09-28 16:40:22.080 UTC [4446] WARNING: there is no transaction in progress
2026-09-28 16:40:22.091 UTC [4446] WARNING: there is no transaction in progress
2026-09-28 16:40:22.115 UTC [4446] WARNING: there is no transaction in progress
2026-09-28 16:40:22.133 UTC [4446] WARNING: there is no transaction in progress
2026-09-28 16:40:22.151 UTC [4446] WARNING: there is no transaction in progress
2026-09-28 16:40:22.173 UTC [4446] WARNING: there is no transaction in progress
2026-09-28 16:40:22.189 UTC [4446] WARNING: there is no transaction in progress
2026-09-28 16:40:22.201 UTC [4446] WARNING: there is no transaction in progress
2026-09-28 16:40:22.213 UTC [4446] WARNING: there is no transaction in progress
2026-09-28 16:40:22.224 UTC [4446] WARNING: there is no transaction in progress
2026-09-28 16:40:23,255 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 16:40:23,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 16:40:23,531 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 16:40:23,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 16:40:23,854 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 16:40:23,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 16:40:24,162 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 16:40:24,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 16:40:24,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 16:40:24,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nqgh.c.gcp-upstream.internal';"
2026-09-28 16:40:24,973 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 16:40:24,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 16:40:25,260 [I] DB_PATCHES have already been applied
I20260928 16:40:25.266954 25885 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ranger-admin:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskp76gbp/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 16:40:25.267083 25885 mini_ranger.cc:193] Using host: 127.25.71.84
I20260928 16:40:25.271613 25885 mini_ranger.cc:227] Ranger admin URL: http://127.25.71.84:33229
Sep 28, 2026 4:40:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:40:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:40:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 4:40:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 4:40:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 4:40:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 4:40:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 16:40:27.331622 25885 mini_ranger.cc:160] Time spent starting Ranger: real 11.464s user 0.000s sys 0.017s
Sep 28, 2026 4:40:51 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 4:40:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 28, 2026 4:40:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 4:40:51 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 4:40:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 4:40:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 4:40:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 4:40:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 4:40:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 4:40:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 4:40:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 4:40:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 4:40:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 4:40:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 4:40:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 4:40:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 4:40:51 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 4:40:52 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 16:40:53.102200 25885 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.25.71.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.25.71.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.25.71.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/rangerkms_127.25.71.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.25.71.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/rangerkms_127.25.71.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.71.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.71.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.71.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/HTTP_127.25.71.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.71.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/HTTP_127.25.71.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 16:40:53 dist-test-slave-nqgh krb5kdc[4233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613653, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 16:40:53.211716 25885 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 16:40:53.225849 25885 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 16:40:54.214123 25885 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 16:40:54.214215 25885 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ranger-kms
I20260928 16:40:54.214340 25885 mini_ranger_kms.cc:227] Using postgres at 127.25.71.84:43533
2026-09-28 16:40:54,870 [I] DB FLAVOR :POSTGRES
2026-09-28 16:40:54,871 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 16:40:54,871 [I] Checking connection
2026-09-28 16:40:54,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 16:40:55,158 [I] connection success
2026-09-28 16:40:55,159 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 16:40:55,159 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 16:40:55,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 16:40:55,466 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 16:40:55,467 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 16:40:55,467 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 16:40:55,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:43533/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 16:40:55,795 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 16:40:56.431190 25885 mini_ranger_kms.cc:310] Created kms service
I20260928 16:40:56.648952 25885 mini_ranger_kms.cc:326] Created kudu user
I20260928 16:40:56.701793 25885 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 16:40:56 dist-test-slave-nqgh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613653, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
I20260928 16:40:57.234134 25885 mini_ranger_kms.cc:384] Added ranger policy
I20260928 16:40:57.234359 25885 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ranger-kms:/tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 16:40:57.234380 25885 mini_ranger_kms.cc:240] Using host: 127.25.71.84
I20260928 16:40:57.239158 25885 mini_ranger_kms.cc:276] Ranger KMS PID: 4628
I20260928 16:40:57.239369 25885 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.25.71.84:51619
Sep 28, 2026 4:40:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:40:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:40:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 4:40:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 4:40:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 4:40:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 16:40:59.373483 25885 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.162s user 0.041s sys 0.273s
I20260928 16:40:59.373651 25885 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 16:40:59.373785 25885 mini_ranger_kms.cc:401] 127.25.71.84:51619/kms/v1/keys
Sep 28 16:40:59 dist-test-slave-nqgh krb5kdc[4233](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790613659, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.71.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 16:41:00 dist-test-slave-nqgh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613659, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.71.84@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
Sep 28 16:41:00 dist-test-slave-nqgh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613659, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.71.84@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
Sep 28 16:41:03 dist-test-slave-nqgh krb5kdc[4233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:41:03.931048 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:42939
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:42939
--ranger_config_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.71.84:51619/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:41:04.074710 4678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:41:04.075140 4678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:41:04.075204 4678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:41:04.079864 4678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:41:04.079977 4678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:41:04.080013 4678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:41:04.080036 4678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:41:04.080070 4678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:41:04.087375 4678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/master-0/wal
--ranger_kms_url=127.25.71.84:51619/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:42939
--ranger_config_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:42939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:41:04.089035 4678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:41:04.090866 4678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:41:04.102483 4683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:41:04.102550 4684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:41:04.102785 4678 server_base.cc:1034] running on GCE node
W20260928 16:41:04.102949 4686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:41:04.103675 4678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:41:04.105201 4678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:41:04.106590 4678 hybrid_clock.cc:648] HybridClock initialized: now 1790613664106533 us; error 65 us; skew 500 ppm
Sep 28 16:41:04 dist-test-slave-nqgh krb5kdc[4233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613664, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:41:04.111181 4678 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:41:04.112721 4678 webserver.cc:460] Webserver started at http://127.25.71.126:34527/ using document root <none> and password file <none>
I20260928 16:41:04.113467 4678 fs_manager.cc:359] Metadata directory not provided
I20260928 16:41:04.113545 4678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:41:04.113811 4678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:41:04 dist-test-slave-nqgh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613664, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
I20260928 16:41:04.189935 4678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "22b1d3c8dd2c4c7481014ad9419a2ccc"
format_stamp: "Formatted at 2026-09-28 16:41:04 on dist-test-slave-nqgh"
server_key: "c9c0241026cae1ad4e0f006fa7c57fe7"
server_key_iv: "40bf17ecd645cc785ddf944f6924e547"
server_key_version: "kuduclusterkey@0"
I20260928 16:41:04.190871 4678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "22b1d3c8dd2c4c7481014ad9419a2ccc"
format_stamp: "Formatted at 2026-09-28 16:41:04 on dist-test-slave-nqgh"
server_key: "c9c0241026cae1ad4e0f006fa7c57fe7"
server_key_iv: "40bf17ecd645cc785ddf944f6924e547"
server_key_version: "kuduclusterkey@0"
I20260928 16:41:04.196429 4678 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:41:04.219679 4693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:41:04.221580 4678 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:41:04.221784 4678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "22b1d3c8dd2c4c7481014ad9419a2ccc"
format_stamp: "Formatted at 2026-09-28 16:41:04 on dist-test-slave-nqgh"
server_key: "c9c0241026cae1ad4e0f006fa7c57fe7"
server_key_iv: "40bf17ecd645cc785ddf944f6924e547"
server_key_version: "kuduclusterkey@0"
I20260928 16:41:04.222018 4678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:41:04.235602 4678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:41:04.240170 4678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:41:04.240418 4678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:41:04.251513 4678 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:42939
I20260928 16:41:04.251590 4745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:42939 every 8 connection(s)
I20260928 16:41:04.252949 4678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:41:04.257089 4746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:41:04.263156 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 4678
I20260928 16:41:04.263319 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:41:04.263680 25885 external_mini_cluster.cc:1428] Setting key e3ea0e3a0ce0cb8764252a458def55cd
I20260928 16:41:04.265399 4746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc: Bootstrap starting.
I20260928 16:41:04.268875 4746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc: Neither blocks nor log segments found. Creating new log.
I20260928 16:41:04.269886 4746 log.cc:824] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc: Log is configured to *not* fsync() on all Append() calls
I20260928 16:41:04.272858 4746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc: No bootstrap required, opened a new log
Sep 28 16:41:04 dist-test-slave-nqgh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:41:04.278175 4746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b1d3c8dd2c4c7481014ad9419a2ccc" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42939 } }
I20260928 16:41:04.278569 4746 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:41:04.278687 4746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b1d3c8dd2c4c7481014ad9419a2ccc, State: Initialized, Role: FOLLOWER
I20260928 16:41:04.279331 4746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b1d3c8dd2c4c7481014ad9419a2ccc" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42939 } }
I20260928 16:41:04.279580 4746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:41:04.279680 4746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:41:04.279814 4746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:41:04.281237 4746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b1d3c8dd2c4c7481014ad9419a2ccc" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42939 } }
I20260928 16:41:04.281736 4746 leader_election.cc:302] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22b1d3c8dd2c4c7481014ad9419a2ccc; no voters:
I20260928 16:41:04.282306 4746 leader_election.cc:288] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:41:04.282510 4751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:41:04.283133 4751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc [term 1 LEADER]: Becoming Leader. State: Replica: 22b1d3c8dd2c4c7481014ad9419a2ccc, State: Running, Role: LEADER
I20260928 16:41:04.283789 4751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b1d3c8dd2c4c7481014ad9419a2ccc" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42939 } }
I20260928 16:41:04.284613 4746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:41:04.285200 4749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:41:04.265759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33446 (local address 127.25.71.126:42939)
0928 16:41:04.266355 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0928 16:41:04.266376 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:41:04.266424 (+ 48us) server_negotiation.cc:407] Connection header received
0928 16:41:04.267351 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:41:04.267376 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:41:04.267854 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:41:04.268285 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:41:04.269319 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:04.270760 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:04.272258 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:04.273061 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:04.273314 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:41:04.276930 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:41:04.276941 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:41:04.276971 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:41:04.277028 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:41:04.279303 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:41:04.280147 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:41:04.280155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:41:04.280164 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:41:04.280362 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:41:04.280738 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:41:04.280740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:41:04.280742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:41:04.281660 (+ 918us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:41:04.282001 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0928 16:41:04.283430 (+ 1429us) server_negotiation.cc:299] Negotiation successful
0928 16:41:04.283867 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":181,"threads_started":1}
I20260928 16:41:04.285655 4751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22b1d3c8dd2c4c7481014ad9419a2ccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b1d3c8dd2c4c7481014ad9419a2ccc" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42939 } } }
I20260928 16:41:04.286019 4751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc [sys.catalog]: This master's current role is: LEADER
I20260928 16:41:04.286329 4751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22b1d3c8dd2c4c7481014ad9419a2ccc. Latest consensus state: current_term: 1 leader_uuid: "22b1d3c8dd2c4c7481014ad9419a2ccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b1d3c8dd2c4c7481014ad9419a2ccc" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42939 } } }
I20260928 16:41:04.286478 4751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc [sys.catalog]: This master's current role is: LEADER
I20260928 16:41:04.286923 4760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:41:04.293303 4760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:41:04.294005 4746 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 16:41:04.310138 4760 catalog_manager.cc:1347] Generated new cluster ID: 72ab8185359b4f1eb082f765b2430767
I20260928 16:41:04.310246 4760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:41:04.404639 4760 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:41:04.406150 4760 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:41:04.421311 4760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc: Generated new TSK 0
I20260928 16:41:04.422457 4760 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
I20260928 16:41:05.320171 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.71.84:51619/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:42939
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:41:05.514570 4791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:41:05.515053 4791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:41:05.515126 4791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:41:05.521777 4791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:41:05.521898 4791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:41:05.521997 4791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:41:05.530966 4791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.25.71.84:51619/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:42939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:41:05.532596 4791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:41:05.534178 4791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:41:05.550268 4798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:41:05.550702 4791 server_base.cc:1034] running on GCE node
W20260928 16:41:05.550443 4801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:41:05.550266 4799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:41:05.551896 4791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:41:05.552835 4791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:41:05.554087 4791 hybrid_clock.cc:648] HybridClock initialized: now 1790613665554060 us; error 46 us; skew 500 ppm
Sep 28 16:41:05 dist-test-slave-nqgh krb5kdc[4233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613665, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:41:05.562307 4791 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:41:05.564013 4791 webserver.cc:460] Webserver started at http://127.25.71.65:41423/ using document root <none> and password file <none>
I20260928 16:41:05.564939 4791 fs_manager.cc:359] Metadata directory not provided
I20260928 16:41:05.565100 4791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:41:05.565394 4791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:41:05 dist-test-slave-nqgh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613665, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
I20260928 16:41:05.586938 4791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "ccdb8a69450a4a0b94b972aa02944572"
format_stamp: "Formatted at 2026-09-28 16:41:05 on dist-test-slave-nqgh"
server_key: "62ae5ea62e0ecc0d2904cf48ea009283"
server_key_iv: "4d11908c51f07abed33cfa1c6ad887ee"
server_key_version: "kuduclusterkey@0"
I20260928 16:41:05.588304 4791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "ccdb8a69450a4a0b94b972aa02944572"
format_stamp: "Formatted at 2026-09-28 16:41:05 on dist-test-slave-nqgh"
server_key: "62ae5ea62e0ecc0d2904cf48ea009283"
server_key_iv: "4d11908c51f07abed33cfa1c6ad887ee"
server_key_version: "kuduclusterkey@0"
I20260928 16:41:05.609354 4791 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.002s sys 0.004s
I20260928 16:41:05.647022 4810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:41:05.652660 4791 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.003s sys 0.001s
I20260928 16:41:05.652916 4791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "ccdb8a69450a4a0b94b972aa02944572"
format_stamp: "Formatted at 2026-09-28 16:41:05 on dist-test-slave-nqgh"
server_key: "62ae5ea62e0ecc0d2904cf48ea009283"
server_key_iv: "4d11908c51f07abed33cfa1c6ad887ee"
server_key_version: "kuduclusterkey@0"
I20260928 16:41:05.653108 4791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:41:05.683348 4791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:41:05.688423 4791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:41:05.688731 4791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:41:05.689612 4791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:41:05.691116 4791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:41:05.691243 4791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 16:41:05.691382 4791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:41:05.691452 4791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:41:05.705196 4791 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:35359
I20260928 16:41:05.705720 4923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:35359 every 8 connection(s)
I20260928 16:41:05.706570 4791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:41:05.707576 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 4791
I20260928 16:41:05.707746 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:41:05.708204 25885 external_mini_cluster.cc:1428] Setting key 4884748c0424e627032ee562c02ab8a9
Sep 28 16:41:05 dist-test-slave-nqgh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613665, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.66@KRBTEST.COM" created.
I20260928 16:41:05.727849 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:41:05.711002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:37315 (local address 127.25.71.126:42939)
0928 16:41:05.711304 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 16:41:05.711309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:41:05.711328 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:41:05.711728 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:41:05.711730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:41:05.711800 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:41:05.711883 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:41:05.713143 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:05.714040 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:05.716033 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:05.716454 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:05.716568 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:41:05.721646 (+ 5078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:41:05.721652 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:41:05.721656 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:41:05.721706 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:41:05.724137 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:41:05.725316 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:41:05.725321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:41:05.725323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:41:05.725486 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:41:05.726047 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:41:05.726051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:41:05.726053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:41:05.726402 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:41:05.726512 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 16:41:05.727444 (+ 932us) server_negotiation.cc:299] Negotiation successful
0928 16:41:05.727625 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":100,"threads_started":1}
I20260928 16:41:05.729674 4926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:41:05.709818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:42939 (local address 127.25.71.65:37315)
0928 16:41:05.710420 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0928 16:41:05.710480 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 16:41:05.711515 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:41:05.712094 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:41:05.712102 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:41:05.712402 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:41:05.712947 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:41:05.712969 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:41:05.714211 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:41:05.714214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:41:05.715841 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:41:05.715857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:41:05.716806 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:41:05.716811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:41:05.717057 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:41:05.717973 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:41:05.717998 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:41:05.721224 (+ 3226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:41:05.724439 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:41:05.724451 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:41:05.724478 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:41:05.725038 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:41:05.725627 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:41:05.725630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:41:05.725632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:41:05.725917 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:41:05.726579 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:41:05.726589 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:41:05.727034 (+ 445us) client_negotiation.cc:769] Sending connection context
0928 16:41:05.727269 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 16:41:05.728434 (+ 1165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":147,"threads_started":1}
Sep 28 16:41:05 dist-test-slave-nqgh krb5kdc[4233](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790613665, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:41:05.731812 4924 heartbeater.cc:347] Connected to a master server at 127.25.71.126:42939
I20260928 16:41:05.732151 4924 heartbeater.cc:464] Registering TS with master...
I20260928 16:41:05.732846 4924 heartbeater.cc:511] Master 127.25.71.126:42939 requested a full tablet report, sending...
I20260928 16:41:05.735502 4710 ts_manager.cc:194] Registered new tserver with Master: ccdb8a69450a4a0b94b972aa02944572 (127.25.71.65:35359)
I20260928 16:41:05.737562 4710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:37315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
I20260928 16:41:05.755748 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.71.84:51619/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:42939
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:41:05.937794 4930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:41:05.938357 4930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:41:05.938478 4930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:41:05.943846 4930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:41:05.944169 4930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:41:05.944392 4930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:41:05.953037 4930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.25.71.84:51619/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:42939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:41:05.954314 4930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:41:05.955696 4930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:41:05.968986 4939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:41:05.968997 4937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:41:05.969578 4936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:41:05.970291 4930 server_base.cc:1034] running on GCE node
I20260928 16:41:05.971016 4930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:41:05.972056 4930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:41:05.973282 4930 hybrid_clock.cc:648] HybridClock initialized: now 1790613665973255 us; error 46 us; skew 500 ppm
Sep 28 16:41:05 dist-test-slave-nqgh krb5kdc[4233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613665, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:41:05.976873 4930 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:41:05.978040 4930 webserver.cc:460] Webserver started at http://127.25.71.66:42413/ using document root <none> and password file <none>
I20260928 16:41:05.978736 4930 fs_manager.cc:359] Metadata directory not provided
I20260928 16:41:05.978832 4930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:41:05.979063 4930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:41:05 dist-test-slave-nqgh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613665, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
I20260928 16:41:06.004422 4930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "ca78ec8efbf84eafa183a0bff7bc78e2"
format_stamp: "Formatted at 2026-09-28 16:41:05 on dist-test-slave-nqgh"
server_key: "700b0a9f4f182c68018c94553c5c6f97"
server_key_iv: "242dfab95a6403515f723f4615e02d4e"
server_key_version: "kuduclusterkey@0"
I20260928 16:41:06.005290 4930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "ca78ec8efbf84eafa183a0bff7bc78e2"
format_stamp: "Formatted at 2026-09-28 16:41:05 on dist-test-slave-nqgh"
server_key: "700b0a9f4f182c68018c94553c5c6f97"
server_key_iv: "242dfab95a6403515f723f4615e02d4e"
server_key_version: "kuduclusterkey@0"
I20260928 16:41:06.012218 4930 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 16:41:06.035070 4946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:41:06.036880 4930 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:41:06.037051 4930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "ca78ec8efbf84eafa183a0bff7bc78e2"
format_stamp: "Formatted at 2026-09-28 16:41:05 on dist-test-slave-nqgh"
server_key: "700b0a9f4f182c68018c94553c5c6f97"
server_key_iv: "242dfab95a6403515f723f4615e02d4e"
server_key_version: "kuduclusterkey@0"
I20260928 16:41:06.037217 4930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:41:06.069200 4930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:41:06.073580 4930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:41:06.073913 4930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:41:06.074753 4930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:41:06.076000 4930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:41:06.076069 4930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:41:06.076128 4930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:41:06.076159 4930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:41:06.090144 4930 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:37563
I20260928 16:41:06.091177 5059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:37563 every 8 connection(s)
I20260928 16:41:06.091856 4930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:41:06.092374 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 4930
I20260928 16:41:06.092525 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:41:06.092885 25885 external_mini_cluster.cc:1428] Setting key 5a2120b5653206422ba6be7f167645bd
WARNING: no policy specified for kudu/127.25.71.67@KRBTEST.COM; defaulting to no policy
Sep 28 16:41:06 dist-test-slave-nqgh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613665, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.67@KRBTEST.COM" created.
I20260928 16:41:06.137765 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:41:06.097688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:42451 (local address 127.25.71.126:42939)
0928 16:41:06.097881 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 16:41:06.097887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:41:06.097912 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:41:06.099175 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:41:06.099180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:41:06.099262 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:41:06.099330 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:41:06.101610 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.102464 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.104722 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.105088 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.105194 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:41:06.131269 (+ 26075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:41:06.131277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:41:06.131280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:41:06.131347 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:41:06.133604 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:41:06.135097 (+ 1493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:41:06.135104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:41:06.135106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:41:06.135247 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:41:06.135851 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:41:06.135856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:41:06.135858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:41:06.136197 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:41:06.136299 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 16:41:06.137300 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0928 16:41:06.137525 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:41:06.141055 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:41:06.096746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:42939 (local address 127.25.71.66:42451)
0928 16:41:06.097466 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0928 16:41:06.097537 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 16:41:06.098789 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:41:06.099935 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:41:06.099948 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:41:06.100439 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:41:06.101346 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:41:06.101374 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.102785 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.102788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:41:06.104521 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:41:06.104538 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.105488 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.105491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:41:06.105762 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:41:06.106725 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:41:06.106752 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:41:06.126955 (+ 20203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:41:06.134074 (+ 7119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:41:06.134090 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:41:06.134123 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:41:06.134798 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:41:06.135413 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:41:06.135417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:41:06.135420 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:41:06.135697 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:41:06.136396 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:41:06.136406 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:41:06.136916 (+ 510us) client_negotiation.cc:769] Sending connection context
0928 16:41:06.138528 (+ 1612us) client_negotiation.cc:240] Negotiation successful
0928 16:41:06.139352 (+ 824us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":187,"threads_started":1}
I20260928 16:41:06.143639 5060 heartbeater.cc:347] Connected to a master server at 127.25.71.126:42939
I20260928 16:41:06.144021 5060 heartbeater.cc:464] Registering TS with master...
I20260928 16:41:06.144934 5060 heartbeater.cc:511] Master 127.25.71.126:42939 requested a full tablet report, sending...
I20260928 16:41:06.147053 4710 ts_manager.cc:194] Registered new tserver with Master: ca78ec8efbf84eafa183a0bff7bc78e2 (127.25.71.66:37563)
I20260928 16:41:06.147962 4710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:42451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
I20260928 16:41:06.187867 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.71.84:51619/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:42939
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:41:06 dist-test-slave-nqgh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613665, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
W20260928 16:41:06.381299 5065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:41:06.381837 5065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:41:06.381970 5065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:41:06.389801 5065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:41:06.389977 5065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:41:06.390085 5065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:41:06.398010 5065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.25.71.84:51619/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:42939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:41:06.399340 5065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:41:06.400516 5065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:41:06.414603 5071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:41:06.414958 5073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:41:06.414613 5070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:41:06.417127 5065 server_base.cc:1034] running on GCE node
I20260928 16:41:06.417968 5065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:41:06.418992 5065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:41:06.422324 5065 hybrid_clock.cc:648] HybridClock initialized: now 1790613666422226 us; error 100 us; skew 500 ppm
Sep 28 16:41:06 dist-test-slave-nqgh krb5kdc[4233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613666, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:41:06.426781 5065 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:41:06.428161 5065 webserver.cc:460] Webserver started at http://127.25.71.67:32945/ using document root <none> and password file <none>
I20260928 16:41:06.429029 5065 fs_manager.cc:359] Metadata directory not provided
I20260928 16:41:06.429194 5065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:41:06.429536 5065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:41:06 dist-test-slave-nqgh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613666, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
I20260928 16:41:06.455686 5065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "4bee7157ce38458b84ae3e6c4aeea6d4"
format_stamp: "Formatted at 2026-09-28 16:41:06 on dist-test-slave-nqgh"
server_key: "ccd02cacbc876d6107407d6a053072b7"
server_key_iv: "314b4aeab5908b507b71e5add2fbe70e"
server_key_version: "kuduclusterkey@0"
I20260928 16:41:06.456630 5065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "4bee7157ce38458b84ae3e6c4aeea6d4"
format_stamp: "Formatted at 2026-09-28 16:41:06 on dist-test-slave-nqgh"
server_key: "ccd02cacbc876d6107407d6a053072b7"
server_key_iv: "314b4aeab5908b507b71e5add2fbe70e"
server_key_version: "kuduclusterkey@0"
I20260928 16:41:06.462833 5065 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:41:06.492658 5080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:41:06.496292 5065 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.001s
I20260928 16:41:06.496490 5065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "4bee7157ce38458b84ae3e6c4aeea6d4"
format_stamp: "Formatted at 2026-09-28 16:41:06 on dist-test-slave-nqgh"
server_key: "ccd02cacbc876d6107407d6a053072b7"
server_key_iv: "314b4aeab5908b507b71e5add2fbe70e"
server_key_version: "kuduclusterkey@0"
I20260928 16:41:06.496800 5065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:41:06.529134 5065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:41:06.533488 5065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:41:06.533819 5065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:41:06.534669 5065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:41:06.539988 5065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:41:06.540081 5065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:41:06.540153 5065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:41:06.540185 5065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:41:06.551537 5065 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:45513
I20260928 16:41:06.552814 5065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:41:06.556079 5193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:45513 every 8 connection(s)
I20260928 16:41:06.561532 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 5065
I20260928 16:41:06.561699 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:41:06.562047 25885 external_mini_cluster.cc:1428] Setting key e6fa068696ad474b2d6a57402f1a589d
Sep 28 16:41:06 dist-test-slave-nqgh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613666, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:41:06.584438 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:41:06.560939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:48787 (local address 127.25.71.126:42939)
0928 16:41:06.561132 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 16:41:06.561137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:41:06.566234 (+ 5097us) server_negotiation.cc:407] Connection header received
0928 16:41:06.567861 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:41:06.567866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:41:06.567941 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:41:06.567994 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:41:06.570253 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.571180 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.573091 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.573476 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.573556 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:41:06.579477 (+ 5921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:41:06.579482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:41:06.579485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:41:06.579523 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:41:06.581318 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:41:06.582173 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:41:06.582177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:41:06.582179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:41:06.582278 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:41:06.582781 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:41:06.582785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:41:06.582787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:41:06.583061 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:41:06.583148 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 16:41:06.584047 (+ 899us) server_negotiation.cc:299] Negotiation successful
0928 16:41:06.584231 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:41:06.586978 5196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:41:06.559379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:42939 (local address 127.25.71.67:48787)
0928 16:41:06.566008 (+ 6629us) negotiation.cc:107] Waiting for socket to connect
0928 16:41:06.566112 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0928 16:41:06.567543 (+ 1431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:41:06.568581 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:41:06.568594 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:41:06.569089 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:41:06.570023 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:41:06.570048 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.571361 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.571365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:41:06.572935 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:41:06.572950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.573840 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.573844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:41:06.574092 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:41:06.574992 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:41:06.575017 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:41:06.579203 (+ 4186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:41:06.581522 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:41:06.581531 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:41:06.581560 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:41:06.582027 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:41:06.582388 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:41:06.582391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:41:06.582394 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:41:06.582616 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:41:06.583242 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:41:06.583251 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:41:06.583704 (+ 453us) client_negotiation.cc:769] Sending connection context
0928 16:41:06.585042 (+ 1338us) client_negotiation.cc:240] Negotiation successful
0928 16:41:06.585540 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6288,"thread_start_us":190,"threads_started":1}
I20260928 16:41:06.590819 5194 heartbeater.cc:347] Connected to a master server at 127.25.71.126:42939
I20260928 16:41:06.591179 5194 heartbeater.cc:464] Registering TS with master...
I20260928 16:41:06.591969 5194 heartbeater.cc:511] Master 127.25.71.126:42939 requested a full tablet report, sending...
I20260928 16:41:06.593554 4710 ts_manager.cc:194] Registered new tserver with Master: 4bee7157ce38458b84ae3e6c4aeea6d4 (127.25.71.67:45513)
I20260928 16:41:06.594401 4710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:48787
I20260928 16:41:06.601459 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:41:06.616171 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:41:06.604987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51658 (local address 127.25.71.126:42939)
0928 16:41:06.605174 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 16:41:06.605181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:41:06.606060 (+ 879us) server_negotiation.cc:407] Connection header received
0928 16:41:06.606390 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:41:06.606394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:41:06.606457 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:41:06.606512 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:41:06.607651 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.608505 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.609722 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.610057 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.610149 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:41:06.612228 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:41:06.612233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:41:06.612236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:41:06.612275 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:41:06.614046 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:41:06.614665 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:41:06.614669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:41:06.614671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:41:06.614767 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:41:06.615051 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:41:06.615054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:41:06.615056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:41:06.615329 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:41:06.615439 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 16:41:06.615806 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 16:41:06.615964 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:41:06.621469 4710 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:41:06.627426 4710 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:41:06.673203 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:41:06.650181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:45513 (local address 127.0.0.1:41284)
0928 16:41:06.655119 (+ 4938us) negotiation.cc:107] Waiting for socket to connect
0928 16:41:06.655135 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:41:06.655312 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:41:06.658403 (+ 3091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:41:06.658412 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:41:06.658489 (+ 77us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:41:06.658743 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:41:06.658751 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.664915 (+ 6164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.664919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:41:06.666349 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:41:06.666367 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.672409 (+ 6042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.672414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:41:06.672608 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:41:06.672661 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 16:41:06.672796 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 16:41:06.672985 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4855,"thread_start_us":160,"threads_started":1}
I20260928 16:41:06.674111 5207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:41:06.651869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:37563 (local address 127.0.0.1:41258)
0928 16:41:06.656872 (+ 5003us) negotiation.cc:107] Waiting for socket to connect
0928 16:41:06.656886 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:41:06.658887 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:41:06.659814 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:41:06.659817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:41:06.659836 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:41:06.659943 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:41:06.659949 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.661428 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.661432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:41:06.667635 (+ 6203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:41:06.667650 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.673656 (+ 6006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.673660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:41:06.673778 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:41:06.673805 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:41:06.673876 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 16:41:06.673965 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4934,"thread_start_us":121,"threads_started":1}
I20260928 16:41:06.676026 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:41:06.647324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:35359 (local address 127.0.0.1:53474)
0928 16:41:06.654474 (+ 7150us) negotiation.cc:107] Waiting for socket to connect
0928 16:41:06.654572 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0928 16:41:06.656656 (+ 2084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:41:06.662171 (+ 5515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:41:06.662176 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:41:06.662208 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:41:06.662427 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:41:06.662438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.672603 (+ 10165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.672608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:41:06.673942 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:41:06.673961 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.675523 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.675527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:41:06.675638 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:41:06.675666 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 16:41:06.675741 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 16:41:06.675843 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6859,"thread_start_us":148,"threads_started":1}
I20260928 16:41:06.676964 5208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:41:06.647531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53474 (local address 127.25.71.65:35359)
0928 16:41:06.661249 (+ 13718us) server_negotiation.cc:206] Beginning negotiation
0928 16:41:06.661261 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:41:06.661309 (+ 48us) server_negotiation.cc:407] Connection header received
0928 16:41:06.661443 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:41:06.661450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:41:06.661731 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:41:06.661947 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:41:06.671143 (+ 9196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.672389 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.674130 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.675136 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.675254 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:41:06.675443 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 16:41:06.676696 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0928 16:41:06.676819 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13545,"thread_start_us":137,"threads_started":1}
I20260928 16:41:06.676975 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:41:06.652404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41258 (local address 127.25.71.66:37563)
0928 16:41:06.654944 (+ 2540us) server_negotiation.cc:206] Beginning negotiation
0928 16:41:06.654955 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:41:06.658268 (+ 3313us) server_negotiation.cc:407] Connection header received
0928 16:41:06.659058 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:41:06.659067 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:41:06.659403 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:41:06.659631 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:41:06.660071 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.661278 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.667846 (+ 6568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.670428 (+ 2582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.670569 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:41:06.670832 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 16:41:06.675142 (+ 4310us) server_negotiation.cc:299] Negotiation successful
0928 16:41:06.675314 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2385,"thread_start_us":126,"threads_started":1}
I20260928 16:41:06.680898 4858 tablet_service.cc:1467] Processing CreateTablet for tablet b913aefebd5d4f17b67e7b66963ec6b3 (DEFAULT_TABLE table=test-table [id=e9f86f22e59c423db7259ea16e2a6fdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:41:06.681021 4993 tablet_service.cc:1467] Processing CreateTablet for tablet b913aefebd5d4f17b67e7b66963ec6b3 (DEFAULT_TABLE table=test-table [id=e9f86f22e59c423db7259ea16e2a6fdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:41:06.682545 4993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b913aefebd5d4f17b67e7b66963ec6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:41:06.682593 4858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b913aefebd5d4f17b67e7b66963ec6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:41:06.697333 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:41:06.650802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41284 (local address 127.25.71.67:45513)
0928 16:41:06.655700 (+ 4898us) server_negotiation.cc:206] Beginning negotiation
0928 16:41:06.655713 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:41:06.655746 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:41:06.655853 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:41:06.655859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:41:06.656208 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:41:06.656424 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:41:06.663408 (+ 6984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.664737 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.670984 (+ 6247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.671986 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.672127 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:41:06.672315 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 16:41:06.675494 (+ 3179us) server_negotiation.cc:299] Negotiation successful
0928 16:41:06.697082 (+ 21588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4761,"thread_start_us":150,"threads_started":1}
I20260928 16:41:06.697863 5213 tablet_bootstrap.cc:492] T b913aefebd5d4f17b67e7b66963ec6b3 P ccdb8a69450a4a0b94b972aa02944572: Bootstrap starting.
I20260928 16:41:06.699760 5214 tablet_bootstrap.cc:492] T b913aefebd5d4f17b67e7b66963ec6b3 P ca78ec8efbf84eafa183a0bff7bc78e2: Bootstrap starting.
I20260928 16:41:06.700877 5213 tablet_bootstrap.cc:654] T b913aefebd5d4f17b67e7b66963ec6b3 P ccdb8a69450a4a0b94b972aa02944572: Neither blocks nor log segments found. Creating new log.
I20260928 16:41:06.701368 5128 tablet_service.cc:1467] Processing CreateTablet for tablet b913aefebd5d4f17b67e7b66963ec6b3 (DEFAULT_TABLE table=test-table [id=e9f86f22e59c423db7259ea16e2a6fdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:41:06.702663 5128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b913aefebd5d4f17b67e7b66963ec6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:41:06.701915 5213 log.cc:824] T b913aefebd5d4f17b67e7b66963ec6b3 P ccdb8a69450a4a0b94b972aa02944572: Log is configured to *not* fsync() on all Append() calls
I20260928 16:41:06.703684 5214 tablet_bootstrap.cc:654] T b913aefebd5d4f17b67e7b66963ec6b3 P ca78ec8efbf84eafa183a0bff7bc78e2: Neither blocks nor log segments found. Creating new log.
I20260928 16:41:06.705425 5214 log.cc:824] T b913aefebd5d4f17b67e7b66963ec6b3 P ca78ec8efbf84eafa183a0bff7bc78e2: Log is configured to *not* fsync() on all Append() calls
I20260928 16:41:06.708848 5214 tablet_bootstrap.cc:492] T b913aefebd5d4f17b67e7b66963ec6b3 P ca78ec8efbf84eafa183a0bff7bc78e2: No bootstrap required, opened a new log
I20260928 16:41:06.709175 5214 ts_tablet_manager.cc:1397] T b913aefebd5d4f17b67e7b66963ec6b3 P ca78ec8efbf84eafa183a0bff7bc78e2: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 16:41:06.716917 5214 raft_consensus.cc:348] T b913aefebd5d4f17b67e7b66963ec6b3 P ca78ec8efbf84eafa183a0bff7bc78e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdb8a69450a4a0b94b972aa02944572" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35359 } } peers { permanent_uuid: "4bee7157ce38458b84ae3e6c4aeea6d4" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 45513 } } peers { permanent_uuid: "ca78ec8efbf84eafa183a0bff7bc78e2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37563 } }
I20260928 16:41:06.717432 5214 raft_consensus.cc:374] T b913aefebd5d4f17b67e7b66963ec6b3 P ca78ec8efbf84eafa183a0bff7bc78e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:41:06.717514 5214 raft_consensus.cc:729] T b913aefebd5d4f17b67e7b66963ec6b3 P ca78ec8efbf84eafa183a0bff7bc78e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca78ec8efbf84eafa183a0bff7bc78e2, State: Initialized, Role: FOLLOWER
I20260928 16:41:06.718314 5214 consensus_queue.cc:260] T b913aefebd5d4f17b67e7b66963ec6b3 P ca78ec8efbf84eafa183a0bff7bc78e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdb8a69450a4a0b94b972aa02944572" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35359 } } peers { permanent_uuid: "4bee7157ce38458b84ae3e6c4aeea6d4" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 45513 } } peers { permanent_uuid: "ca78ec8efbf84eafa183a0bff7bc78e2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37563 } }
I20260928 16:41:06.719923 5214 ts_tablet_manager.cc:1428] T b913aefebd5d4f17b67e7b66963ec6b3 P ca78ec8efbf84eafa183a0bff7bc78e2: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260928 16:41:06.720453 5060 heartbeater.cc:503] Master 127.25.71.126:42939 was elected leader, sending a full tablet report...
I20260928 16:41:06.727276 5217 tablet_bootstrap.cc:492] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4: Bootstrap starting.
I20260928 16:41:06.729944 5213 tablet_bootstrap.cc:492] T b913aefebd5d4f17b67e7b66963ec6b3 P ccdb8a69450a4a0b94b972aa02944572: No bootstrap required, opened a new log
I20260928 16:41:06.730309 5213 ts_tablet_manager.cc:1397] T b913aefebd5d4f17b67e7b66963ec6b3 P ccdb8a69450a4a0b94b972aa02944572: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.000s
I20260928 16:41:06.731441 5217 tablet_bootstrap.cc:654] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4: Neither blocks nor log segments found. Creating new log.
I20260928 16:41:06.732743 5217 log.cc:824] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4: Log is configured to *not* fsync() on all Append() calls
I20260928 16:41:06.734867 5213 raft_consensus.cc:348] T b913aefebd5d4f17b67e7b66963ec6b3 P ccdb8a69450a4a0b94b972aa02944572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdb8a69450a4a0b94b972aa02944572" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35359 } } peers { permanent_uuid: "4bee7157ce38458b84ae3e6c4aeea6d4" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 45513 } } peers { permanent_uuid: "ca78ec8efbf84eafa183a0bff7bc78e2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37563 } }
I20260928 16:41:06.735309 5213 raft_consensus.cc:374] T b913aefebd5d4f17b67e7b66963ec6b3 P ccdb8a69450a4a0b94b972aa02944572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:41:06.735430 5213 raft_consensus.cc:729] T b913aefebd5d4f17b67e7b66963ec6b3 P ccdb8a69450a4a0b94b972aa02944572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccdb8a69450a4a0b94b972aa02944572, State: Initialized, Role: FOLLOWER
I20260928 16:41:06.736064 5213 consensus_queue.cc:260] T b913aefebd5d4f17b67e7b66963ec6b3 P ccdb8a69450a4a0b94b972aa02944572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdb8a69450a4a0b94b972aa02944572" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35359 } } peers { permanent_uuid: "4bee7157ce38458b84ae3e6c4aeea6d4" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 45513 } } peers { permanent_uuid: "ca78ec8efbf84eafa183a0bff7bc78e2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37563 } }
I20260928 16:41:06.737349 5213 ts_tablet_manager.cc:1428] T b913aefebd5d4f17b67e7b66963ec6b3 P ccdb8a69450a4a0b94b972aa02944572: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:41:06.737787 4924 heartbeater.cc:503] Master 127.25.71.126:42939 was elected leader, sending a full tablet report...
I20260928 16:41:06.744318 5217 tablet_bootstrap.cc:492] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4: No bootstrap required, opened a new log
I20260928 16:41:06.744670 5217 ts_tablet_manager.cc:1397] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.004s
I20260928 16:41:06.756819 5217 raft_consensus.cc:348] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdb8a69450a4a0b94b972aa02944572" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35359 } } peers { permanent_uuid: "4bee7157ce38458b84ae3e6c4aeea6d4" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 45513 } } peers { permanent_uuid: "ca78ec8efbf84eafa183a0bff7bc78e2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37563 } }
I20260928 16:41:06.757393 5217 raft_consensus.cc:374] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:41:06.757471 5217 raft_consensus.cc:729] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bee7157ce38458b84ae3e6c4aeea6d4, State: Initialized, Role: FOLLOWER
I20260928 16:41:06.758320 5217 consensus_queue.cc:260] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdb8a69450a4a0b94b972aa02944572" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35359 } } peers { permanent_uuid: "4bee7157ce38458b84ae3e6c4aeea6d4" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 45513 } } peers { permanent_uuid: "ca78ec8efbf84eafa183a0bff7bc78e2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37563 } }
I20260928 16:41:06.763329 5217 ts_tablet_manager.cc:1428] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4: Time spent starting tablet: real 0.019s user 0.010s sys 0.001s
I20260928 16:41:06.763953 5194 heartbeater.cc:503] Master 127.25.71.126:42939 was elected leader, sending a full tablet report...
Sep 28 16:41:06 dist-test-slave-nqgh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613665, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
W20260928 16:41:06.816043 5195 tablet.cc:2369] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:41:06.844202 5061 tablet.cc:2369] T b913aefebd5d4f17b67e7b66963ec6b3 P ca78ec8efbf84eafa183a0bff7bc78e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:41:06.867839 5221 raft_consensus.cc:482] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:41:06.868232 5221 raft_consensus.cc:504] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdb8a69450a4a0b94b972aa02944572" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35359 } } peers { permanent_uuid: "4bee7157ce38458b84ae3e6c4aeea6d4" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 45513 } } peers { permanent_uuid: "ca78ec8efbf84eafa183a0bff7bc78e2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37563 } }
I20260928 16:41:06.870165 5221 leader_election.cc:288] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccdb8a69450a4a0b94b972aa02944572 (127.25.71.65:35359), ca78ec8efbf84eafa183a0bff7bc78e2 (127.25.71.66:37563)
I20260928 16:41:06.877223 5196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:41:06.871459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:35359 (local address 127.25.71.67:54167)
0928 16:41:06.871649 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 16:41:06.871668 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:41:06.871849 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:41:06.872371 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:41:06.872374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:41:06.872399 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:41:06.872644 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:41:06.872656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.873912 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.873916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:41:06.875419 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:41:06.875435 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.876547 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.876551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:41:06.876796 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:41:06.876828 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 16:41:06.876913 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 16:41:06.877026 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 16:41:06.877841 5208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:41:06.871146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:54167 (local address 127.25.71.65:35359)
0928 16:41:06.871322 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 16:41:06.871327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:41:06.871992 (+ 665us) server_negotiation.cc:407] Connection header received
0928 16:41:06.872073 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:41:06.872076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:41:06.872139 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:41:06.872210 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:41:06.872911 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.873767 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.875601 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.876209 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.876344 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:41:06.876413 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 16:41:06.877596 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0928 16:41:06.877709 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:41:06.878803 4876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b913aefebd5d4f17b67e7b66963ec6b3" candidate_uuid: "4bee7157ce38458b84ae3e6c4aeea6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccdb8a69450a4a0b94b972aa02944572" is_pre_election: true
I20260928 16:41:06.879242 4876 raft_consensus.cc:2454] T b913aefebd5d4f17b67e7b66963ec6b3 P ccdb8a69450a4a0b94b972aa02944572 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bee7157ce38458b84ae3e6c4aeea6d4 in term 0.
I20260928 16:41:06.880162 5084 leader_election.cc:302] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bee7157ce38458b84ae3e6c4aeea6d4, ccdb8a69450a4a0b94b972aa02944572; no voters:
I20260928 16:41:06.880568 5221 raft_consensus.cc:2781] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:41:06.880648 5221 raft_consensus.cc:482] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:41:06.880685 5221 raft_consensus.cc:3037] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:41:06.882346 5221 raft_consensus.cc:504] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdb8a69450a4a0b94b972aa02944572" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35359 } } peers { permanent_uuid: "4bee7157ce38458b84ae3e6c4aeea6d4" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 45513 } } peers { permanent_uuid: "ca78ec8efbf84eafa183a0bff7bc78e2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37563 } }
I20260928 16:41:06.882920 5221 leader_election.cc:288] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 [CANDIDATE]: Term 1 election: Requested vote from peers ccdb8a69450a4a0b94b972aa02944572 (127.25.71.65:35359), ca78ec8efbf84eafa183a0bff7bc78e2 (127.25.71.66:37563)
I20260928 16:41:06.885260 4876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b913aefebd5d4f17b67e7b66963ec6b3" candidate_uuid: "4bee7157ce38458b84ae3e6c4aeea6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccdb8a69450a4a0b94b972aa02944572"
I20260928 16:41:06.885499 4876 raft_consensus.cc:3037] T b913aefebd5d4f17b67e7b66963ec6b3 P ccdb8a69450a4a0b94b972aa02944572 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:41:06.887432 4876 raft_consensus.cc:2454] T b913aefebd5d4f17b67e7b66963ec6b3 P ccdb8a69450a4a0b94b972aa02944572 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bee7157ce38458b84ae3e6c4aeea6d4 in term 1.
I20260928 16:41:06.888377 5084 leader_election.cc:302] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bee7157ce38458b84ae3e6c4aeea6d4, ccdb8a69450a4a0b94b972aa02944572; no voters:
I20260928 16:41:06.888727 5221 raft_consensus.cc:2781] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:41:06.889040 5221 raft_consensus.cc:686] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 [term 1 LEADER]: Becoming Leader. State: Replica: 4bee7157ce38458b84ae3e6c4aeea6d4, State: Running, Role: LEADER
I20260928 16:41:06.889554 5221 consensus_queue.cc:237] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdb8a69450a4a0b94b972aa02944572" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35359 } } peers { permanent_uuid: "4bee7157ce38458b84ae3e6c4aeea6d4" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 45513 } } peers { permanent_uuid: "ca78ec8efbf84eafa183a0bff7bc78e2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37563 } }
I20260928 16:41:06.900442 4710 catalog_manager.cc:5495] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bee7157ce38458b84ae3e6c4aeea6d4 (127.25.71.67). New cstate: current_term: 1 leader_uuid: "4bee7157ce38458b84ae3e6c4aeea6d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdb8a69450a4a0b94b972aa02944572" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bee7157ce38458b84ae3e6c4aeea6d4" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 45513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca78ec8efbf84eafa183a0bff7bc78e2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37563 } health_report { overall_health: UNKNOWN } } }
I20260928 16:41:06.915540 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:41:06.872413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:55243 (local address 127.25.71.66:37563)
0928 16:41:06.872845 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 16:41:06.872850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:41:06.879635 (+ 6785us) server_negotiation.cc:407] Connection header received
0928 16:41:06.906790 (+ 27155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:41:06.906796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:41:06.906883 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:41:06.906937 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:41:06.907494 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.908203 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.910717 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.911382 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.911498 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:41:06.911561 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 16:41:06.915236 (+ 3675us) server_negotiation.cc:299] Negotiation successful
0928 16:41:06.915359 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333}
I20260928 16:41:06.916342 5224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:41:06.871843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:37563 (local address 127.25.71.67:55243)
0928 16:41:06.877823 (+ 5980us) negotiation.cc:107] Waiting for socket to connect
0928 16:41:06.877840 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:41:06.905567 (+ 27727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:41:06.907151 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:41:06.907154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:41:06.907175 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:41:06.907363 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:41:06.907371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.908370 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.908373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:41:06.909552 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:41:06.909566 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.914897 (+ 5331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.914902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:41:06.915046 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:41:06.915080 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 16:41:06.916035 (+ 955us) client_negotiation.cc:240] Negotiation successful
0928 16:41:06.916146 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5893,"thread_start_us":196,"threads_started":1}
I20260928 16:41:06.917477 4996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b913aefebd5d4f17b67e7b66963ec6b3" candidate_uuid: "4bee7157ce38458b84ae3e6c4aeea6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca78ec8efbf84eafa183a0bff7bc78e2" is_pre_election: true
I20260928 16:41:06.918007 4996 raft_consensus.cc:2454] T b913aefebd5d4f17b67e7b66963ec6b3 P ca78ec8efbf84eafa183a0bff7bc78e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bee7157ce38458b84ae3e6c4aeea6d4 in term 0.
I20260928 16:41:06.918555 5002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b913aefebd5d4f17b67e7b66963ec6b3" candidate_uuid: "4bee7157ce38458b84ae3e6c4aeea6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca78ec8efbf84eafa183a0bff7bc78e2"
I20260928 16:41:06.919476 5002 raft_consensus.cc:3037] T b913aefebd5d4f17b67e7b66963ec6b3 P ca78ec8efbf84eafa183a0bff7bc78e2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:41:06.921281 5002 raft_consensus.cc:2454] T b913aefebd5d4f17b67e7b66963ec6b3 P ca78ec8efbf84eafa183a0bff7bc78e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bee7157ce38458b84ae3e6c4aeea6d4 in term 1.
I20260928 16:41:06.962996 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:41:06.956364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.25.71.67:45513)
0928 16:41:06.956558 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 16:41:06.956563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:41:06.956674 (+ 111us) server_negotiation.cc:407] Connection header received
0928 16:41:06.956941 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:41:06.956945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:41:06.957165 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:41:06.957302 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:41:06.957805 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.958726 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.959973 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:41:06.960326 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:41:06.960462 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:41:06.961078 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:41:06.961889 (+ 811us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:41:06.962028 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 16:41:06.962700 (+ 672us) server_negotiation.cc:299] Negotiation successful
0928 16:41:06.962811 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260928 16:41:06.965164 4925 tablet.cc:2369] T b913aefebd5d4f17b67e7b66963ec6b3 P ccdb8a69450a4a0b94b972aa02944572: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:41:06.972930 5002 raft_consensus.cc:1261] T b913aefebd5d4f17b67e7b66963ec6b3 P ca78ec8efbf84eafa183a0bff7bc78e2 [term 1 FOLLOWER]: Refusing update from remote peer 4bee7157ce38458b84ae3e6c4aeea6d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:41:06.974079 5225 consensus_queue.cc:1035] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca78ec8efbf84eafa183a0bff7bc78e2" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:41:06.981823 4876 raft_consensus.cc:1261] T b913aefebd5d4f17b67e7b66963ec6b3 P ccdb8a69450a4a0b94b972aa02944572 [term 1 FOLLOWER]: Refusing update from remote peer 4bee7157ce38458b84ae3e6c4aeea6d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:41:06.982806 5225 consensus_queue.cc:1035] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccdb8a69450a4a0b94b972aa02944572" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:41:06.994408 5228 mvcc.cc:205] Tried to move back new op lower bound from 7334353579908304896 to 7334353579589201920. Current Snapshot: MvccSnapshot[applied={T|T < 7334353579908304896}]
I20260928 16:41:07.022341 4709 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:41:07.022588 4709 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:41:07.026697 4709 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22b1d3c8dd2c4c7481014ad9419a2ccc: Sending DeleteTablet for 3 replicas of tablet b913aefebd5d4f17b67e7b66963ec6b3
I20260928 16:41:07.028460 4993 tablet_service.cc:1514] Processing DeleteTablet for tablet b913aefebd5d4f17b67e7b66963ec6b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:41:07 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:41258
I20260928 16:41:07.029151 5128 tablet_service.cc:1514] Processing DeleteTablet for tablet b913aefebd5d4f17b67e7b66963ec6b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:41:07 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:41284
I20260928 16:41:07.030045 4858 tablet_service.cc:1514] Processing DeleteTablet for tablet b913aefebd5d4f17b67e7b66963ec6b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:41:07 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:53474
I20260928 16:41:07.031344 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 4791
I20260928 16:41:07.035255 5238 tablet_replica.cc:330] T b913aefebd5d4f17b67e7b66963ec6b3 P ca78ec8efbf84eafa183a0bff7bc78e2: stopping tablet replica
I20260928 16:41:07.035555 5239 tablet_replica.cc:330] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4: stopping tablet replica
I20260928 16:41:07.039425 5239 raft_consensus.cc:2229] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:41:07.039933 5238 raft_consensus.cc:2229] T b913aefebd5d4f17b67e7b66963ec6b3 P ca78ec8efbf84eafa183a0bff7bc78e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:41:07.040437 5238 raft_consensus.cc:2258] T b913aefebd5d4f17b67e7b66963ec6b3 P ca78ec8efbf84eafa183a0bff7bc78e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:41:07.040984 5239 raft_consensus.cc:2258] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:41:07.044119 5239 ts_tablet_manager.cc:1905] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:41:07.044764 5238 ts_tablet_manager.cc:1905] T b913aefebd5d4f17b67e7b66963ec6b3 P ca78ec8efbf84eafa183a0bff7bc78e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:41:07.048087 5238 ts_tablet_manager.cc:1918] T b913aefebd5d4f17b67e7b66963ec6b3 P ca78ec8efbf84eafa183a0bff7bc78e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:41:07.053009 5239 ts_tablet_manager.cc:1918] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260928 16:41:07.054450 4697 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.71.65:35359 (error 108)
I20260928 16:41:07.055411 5239 log.cc:1196] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/b913aefebd5d4f17b67e7b66963ec6b3
I20260928 16:41:07.055722 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 4930
I20260928 16:41:07.063197 5239 ts_tablet_manager.cc:1939] T b913aefebd5d4f17b67e7b66963ec6b3 P 4bee7157ce38458b84ae3e6c4aeea6d4: Deleting consensus metadata
I20260928 16:41:07.064925 4697 catalog_manager.cc:4841] TS 4bee7157ce38458b84ae3e6c4aeea6d4 (127.25.71.67:45513): tablet b913aefebd5d4f17b67e7b66963ec6b3 (table test-table [id=e9f86f22e59c423db7259ea16e2a6fdc]) successfully deleted
I20260928 16:41:07.067819 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:41:07.067419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:35359 (local address 127.0.0.1:53480)
0928 16:41:07.067631 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 16:41:07.067714 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.65:35359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260928 16:41:07.068379 4697 catalog_manager.cc:4584] TS ccdb8a69450a4a0b94b972aa02944572 (127.25.71.65:35359): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b913aefebd5d4f17b67e7b66963ec6b3: Network error: Client connection negotiation failed: client connection to 127.25.71.65:35359: connect: Connection refused (error 111)
W20260928 16:41:07.070199 5211 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:41:07.069262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:37563 (local address 127.0.0.1:41266)
0928 16:41:07.069512 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 16:41:07.069537 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 16:41:07.069655 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:41:07.070068 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.66:37563: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 16:41:07.070989 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 5065
I20260928 16:41:07.081404 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 4678
I20260928 16:41:07.089751 25885 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 16:41:07.436981 25885 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 16:41:07.437136 25885 mini_ranger.cc:66] Stopping Ranger...
I20260928 16:41:07.783777 25885 mini_ranger.cc:68] Stopped Ranger
2026-09-28 16:41:07.784 UTC [4263] LOG: received smart shutdown request
2026-09-28 16:41:07.789 UTC [4263] LOG: background worker "logical replication launcher" (PID 4272) exited with exit code 1
2026-09-28 16:41:07.789 UTC [4267] LOG: shutting down
2026-09-28 16:41:08.012 UTC [4263] LOG: database system is shut down
2026-09-28T16:41:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (57577 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:41:08 dist-test-slave-nqgh krb5kdc[5250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:41:08 dist-test-slave-nqgh krb5kdc[5250](info): set up 2 sockets
Sep 28 16:41:08 dist-test-slave-nqgh krb5kdc[5250](info): commencing operation
krb5kdc: starting...
W20260928 16:41:09.104722 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:41:09 dist-test-slave-nqgh krb5kdc[5250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:41:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:41:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.25.71.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.25.71.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.25.71.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/rangeradmin_127.25.71.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.25.71.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/rangeradmin_127.25.71.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.25.71.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.25.71.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.25.71.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/rangerlookup_127.25.71.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.25.71.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/rangerlookup_127.25.71.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.71.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.71.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.71.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/HTTP_127.25.71.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.71.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/HTTP_127.25.71.84@KRBTEST.COM.keytab.
I20260928 16:41:09.327329 25885 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskp76gbp/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/postgres -l logfile start
2026-09-28 16:41:12.461 UTC [5280] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 16:41:12.461 UTC [5280] LOG: listening on IPv4 address "127.25.71.84", port 59553
2026-09-28 16:41:12.466 UTC [5280] LOG: listening on Unix socket "/tmp/.s.PGSQL.59553"
2026-09-28 16:41:12.488 UTC [5283] LOG: database system was shut down at 2026-09-28 16:41:10 UTC
2026-09-28 16:41:12.492 UTC [5280] LOG: database system is ready to accept connections
I20260928 16:41:13.364333 25885 mini_postgres.cc:96] Postgres bound to 59553
2026-09-28 16:41:13.371 UTC [5291] FATAL: database "slave" does not exist
127.25.71.84:59553 - accepting connections
I20260928 16:41:13.372316 25885 mini_ranger.cc:161] Starting Ranger...
I20260928 16:41:13.389870 25885 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 16:41:14.036707 25885 mini_ranger.cc:87] Created ranger Postgres database
I20260928 16:41:14.036831 25885 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ranger-admin
2026-09-28 16:41:14,349 [I] DB FLAVOR :POSTGRES
2026-09-28 16:41:14,349 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 16:41:14,349 [I] Checking connection..
2026-09-28 16:41:14,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 16:41:14,629 [I] Checking connection passed.
2026-09-28 16:41:14,629 [I] --------- Verifying version history table ---------
2026-09-28 16:41:14,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:41:14,909 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 16:41:14,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:41:15,213 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 16:41:15,213 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 16:41:15,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 16:41:15.499 UTC [5355] WARNING: there is no transaction in progress
2026-09-28 16:41:15,508 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 16:41:15,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:41:15,808 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 16:41:15,808 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 16:41:15,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 16:41:16,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 16:41:16,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nqgh.c.gcp-upstream.internal','N') ;"
2026-09-28 16:41:16,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 16:41:16,939 [I] Table x_portal_user does not exist in database ranger
2026-09-28 16:41:16,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 16:41:17,236 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 16:41:17,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 16:41:17,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 16:41:17,815 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 16:41:17,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 16:41:19.132 UTC [5463] WARNING: there is no transaction in progress
2026-09-28 16:41:19.156 UTC [5463] WARNING: there is no transaction in progress
2026-09-28 16:41:19.180 UTC [5463] WARNING: there is no transaction in progress
2026-09-28 16:41:19.203 UTC [5463] WARNING: there is no transaction in progress
2026-09-28 16:41:19.226 UTC [5463] WARNING: there is no transaction in progress
2026-09-28 16:41:19.251 UTC [5463] WARNING: there is no transaction in progress
2026-09-28 16:41:19.328 UTC [5463] WARNING: there is no transaction in progress
2026-09-28 16:41:19.339 UTC [5463] WARNING: there is no transaction in progress
2026-09-28 16:41:19.361 UTC [5463] WARNING: there is no transaction in progress
2026-09-28 16:41:19.376 UTC [5463] WARNING: there is no transaction in progress
2026-09-28 16:41:19.390 UTC [5463] WARNING: there is no transaction in progress
2026-09-28 16:41:19.410 UTC [5463] WARNING: there is no transaction in progress
2026-09-28 16:41:19.424 UTC [5463] WARNING: there is no transaction in progress
2026-09-28 16:41:19.434 UTC [5463] WARNING: there is no transaction in progress
2026-09-28 16:41:19.444 UTC [5463] WARNING: there is no transaction in progress
2026-09-28 16:41:19.456 UTC [5463] WARNING: there is no transaction in progress
2026-09-28 16:41:20,473 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 16:41:20,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 16:41:20,732 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 16:41:20,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 16:41:21,007 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 16:41:21,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 16:41:21,314 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 16:41:21,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 16:41:21,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 16:41:21,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nqgh.c.gcp-upstream.internal';"
2026-09-28 16:41:22,183 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 16:41:22,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 16:41:22,510 [I] DB_PATCHES have already been applied
I20260928 16:41:22.516518 25885 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ranger-admin:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskp76gbp/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 16:41:22.516831 25885 mini_ranger.cc:193] Using host: 127.25.71.84
I20260928 16:41:22.521603 25885 mini_ranger.cc:227] Ranger admin URL: http://127.25.71.84:35319
Sep 28, 2026 4:41:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:41:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:41:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 4:41:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 4:41:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 4:41:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 4:41:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 16:41:23.565536 25885 mini_ranger.cc:160] Time spent starting Ranger: real 10.193s user 0.000s sys 0.013s
Sep 28, 2026 4:41:48 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 4:41:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 4:41:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 4:41:48 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 4:41:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 4:41:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 4:41:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 4:41:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 4:41:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 4:41:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 4:41:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 4:41:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 4:41:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 4:41:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 4:41:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 4:41:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 4:41:48 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 4:41:50 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 16:41:50.737756 25885 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.25.71.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.25.71.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.25.71.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/rangerkms_127.25.71.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.25.71.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/rangerkms_127.25.71.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.71.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.71.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.71.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/HTTP_127.25.71.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.71.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/HTTP_127.25.71.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 16:41:50 dist-test-slave-nqgh krb5kdc[5250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613710, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 16:41:50.861033 25885 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 16:41:50.875399 25885 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 16:41:52.154784 25885 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 16:41:52.154908 25885 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ranger-kms
I20260928 16:41:52.154928 25885 mini_ranger_kms.cc:227] Using postgres at 127.25.71.84:59553
2026-09-28 16:41:52,367 [I] DB FLAVOR :POSTGRES
2026-09-28 16:41:52,367 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 16:41:52,367 [I] Checking connection
2026-09-28 16:41:52,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 16:41:52,659 [I] connection success
2026-09-28 16:41:52,660 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 16:41:52,660 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 16:41:52,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 16:41:52,937 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 16:41:52,938 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 16:41:52,938 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 16:41:52,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.71.84:59553/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 16:41:53,259 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 16:41:53.880492 25885 mini_ranger_kms.cc:310] Created kms service
I20260928 16:41:54.084836 25885 mini_ranger_kms.cc:326] Created kudu user
I20260928 16:41:54.156836 25885 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 16:41:54 dist-test-slave-nqgh krb5kdc[5250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613710, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
I20260928 16:41:54.649747 25885 mini_ranger_kms.cc:384] Added ranger policy
I20260928 16:41:54.649971 25885 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ranger-kms:/tmp/dist-test-taskp76gbp/build/debug/bin/postgresql.jar:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskp76gbp/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 16:41:54.650000 25885 mini_ranger_kms.cc:240] Using host: 127.25.71.84
I20260928 16:41:54.655328 25885 mini_ranger_kms.cc:276] Ranger KMS PID: 5644
I20260928 16:41:54.655478 25885 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.25.71.84:45447
Sep 28, 2026 4:41:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:41:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:41:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 4:41:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 4:41:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 4:41:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 16:41:56.788923 25885 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.928s user 0.008s sys 0.189s
I20260928 16:41:56.789093 25885 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 16:41:56.789165 25885 mini_ranger_kms.cc:401] 127.25.71.84:45447/kms/v1/keys
Sep 28 16:41:57 dist-test-slave-nqgh krb5kdc[5250](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790613717, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.71.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 16:41:57 dist-test-slave-nqgh krb5kdc[5250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613717, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.71.84@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
Sep 28 16:41:58 dist-test-slave-nqgh krb5kdc[5250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613717, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.71.84@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
Sep 28 16:42:01 dist-test-slave-nqgh krb5kdc[5250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:42:01.079725 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:35713
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:35713
--ranger_config_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.71.84:45447/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:42:01.226125 5694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:01.226539 5694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:01.226585 5694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:01.231575 5694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:42:01.231693 5694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:01.231714 5694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:01.231729 5694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:42:01.231745 5694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:42:01.238986 5694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/wal
--ranger_kms_url=127.25.71.84:45447/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:35713
--ranger_config_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:35713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:01.240609 5694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:01.242024 5694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:01.254700 5702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:01.254668 5700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:01.255254 5694 server_base.cc:1034] running on GCE node
W20260928 16:42:01.254731 5699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:01.256420 5694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:01.258157 5694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:01.259469 5694 hybrid_clock.cc:648] HybridClock initialized: now 1790613721259439 us; error 61 us; skew 500 ppm
Sep 28 16:42:01 dist-test-slave-nqgh krb5kdc[5250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613721, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:01.263715 5694 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:42:01.265166 5694 webserver.cc:460] Webserver started at http://127.25.71.126:39619/ using document root <none> and password file <none>
I20260928 16:42:01.265916 5694 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:01.265990 5694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:01.266271 5694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:42:01 dist-test-slave-nqgh krb5kdc[5250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613721, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
I20260928 16:42:01.339978 5694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "40d30c200c4b4935bfa2bdedd1edf9a2"
format_stamp: "Formatted at 2026-09-28 16:42:01 on dist-test-slave-nqgh"
server_key: "d1461ec2f29eb8e8cbe40463dc5fd529"
server_key_iv: "300b3cef02bb76d2b8a6dd7b7f29d24f"
server_key_version: "kuduclusterkey@0"
I20260928 16:42:01.340780 5694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "40d30c200c4b4935bfa2bdedd1edf9a2"
format_stamp: "Formatted at 2026-09-28 16:42:01 on dist-test-slave-nqgh"
server_key: "d1461ec2f29eb8e8cbe40463dc5fd529"
server_key_iv: "300b3cef02bb76d2b8a6dd7b7f29d24f"
server_key_version: "kuduclusterkey@0"
I20260928 16:42:01.345991 5694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 16:42:01.369302 5709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:01.371510 5694 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:42:01.371716 5694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "40d30c200c4b4935bfa2bdedd1edf9a2"
format_stamp: "Formatted at 2026-09-28 16:42:01 on dist-test-slave-nqgh"
server_key: "d1461ec2f29eb8e8cbe40463dc5fd529"
server_key_iv: "300b3cef02bb76d2b8a6dd7b7f29d24f"
server_key_version: "kuduclusterkey@0"
I20260928 16:42:01.371899 5694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:01.403282 5694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:01.407044 5694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:01.407258 5694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:01.416970 5694 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:35713
I20260928 16:42:01.417045 5761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:35713 every 8 connection(s)
I20260928 16:42:01.418545 5694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:42:01.421609 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 5694
I20260928 16:42:01.421762 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:42:01.422104 25885 external_mini_cluster.cc:1428] Setting key fb6c34e8d8b492c2e1ce2e49f675ff03
I20260928 16:42:01.423409 5762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:01.432958 5762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2: Bootstrap starting.
Sep 28 16:42:01 dist-test-slave-nqgh krb5kdc[5250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:01.437000 5762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2: Neither blocks nor log segments found. Creating new log.
I20260928 16:42:01.438551 5762 log.cc:824] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:01.441990 5762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2: No bootstrap required, opened a new log
I20260928 16:42:01.447005 5762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d30c200c4b4935bfa2bdedd1edf9a2" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35713 } }
I20260928 16:42:01.447458 5762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:01.447583 5762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d30c200c4b4935bfa2bdedd1edf9a2, State: Initialized, Role: FOLLOWER
I20260928 16:42:01.448231 5762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d30c200c4b4935bfa2bdedd1edf9a2" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35713 } }
I20260928 16:42:01.448429 5762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:42:01.448504 5762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:42:01.448611 5762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:01.448611 5765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:01.424327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54728 (local address 127.25.71.126:35713)
0928 16:42:01.425095 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:01.425123 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:01.425191 (+ 68us) server_negotiation.cc:407] Connection header received
0928 16:42:01.426295 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:01.426332 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:01.426974 (+ 642us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:01.427468 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:01.428838 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:01.430578 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:01.432232 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:01.433127 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:01.433446 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:01.437601 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:01.437612 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:01.437639 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:01.437707 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:01.443578 (+ 5871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:01.444504 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:01.444512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:01.444522 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:01.444736 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:01.445252 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:01.445257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:01.445259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:01.446115 (+ 856us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:01.446386 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:01.446928 (+ 542us) server_negotiation.cc:299] Negotiation successful
0928 16:42:01.447361 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":220,"threads_started":1}
I20260928 16:42:01.450165 5762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d30c200c4b4935bfa2bdedd1edf9a2" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35713 } }
I20260928 16:42:01.450739 5762 leader_election.cc:302] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40d30c200c4b4935bfa2bdedd1edf9a2; no voters:
I20260928 16:42:01.451226 5762 leader_election.cc:288] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:42:01.451411 5767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:42:01.451879 5767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [term 1 LEADER]: Becoming Leader. State: Replica: 40d30c200c4b4935bfa2bdedd1edf9a2, State: Running, Role: LEADER
I20260928 16:42:01.452457 5767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d30c200c4b4935bfa2bdedd1edf9a2" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35713 } }
I20260928 16:42:01.453042 5762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:42:01.454226 5768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40d30c200c4b4935bfa2bdedd1edf9a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d30c200c4b4935bfa2bdedd1edf9a2" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35713 } } }
I20260928 16:42:01.454301 5769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40d30c200c4b4935bfa2bdedd1edf9a2. Latest consensus state: current_term: 1 leader_uuid: "40d30c200c4b4935bfa2bdedd1edf9a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d30c200c4b4935bfa2bdedd1edf9a2" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35713 } } }
I20260928 16:42:01.454485 5769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [sys.catalog]: This master's current role is: LEADER
I20260928 16:42:01.454423 5768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [sys.catalog]: This master's current role is: LEADER
I20260928 16:42:01.455379 5776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:42:01.455852 5762 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 16:42:01.460496 5776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:42:01.475570 5776 catalog_manager.cc:1347] Generated new cluster ID: 6f7f66d4dea64769b456e2881edc8e0a
I20260928 16:42:01.475716 5776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:42:01.491041 5776 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:42:01.492782 5776 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:42:01.501870 5776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2: Generated new TSK 0
I20260928 16:42:01.503491 5776 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
I20260928 16:42:02.421155 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.71.84:45447/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:35713
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:42:02.619428 5807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:02.620101 5807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:02.620224 5807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:02.627691 5807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:02.627836 5807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:02.627940 5807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:42:02.637205 5807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.25.71.84:45447/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:35713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:02.638892 5807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:02.640339 5807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:02.653165 5815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:02.653206 5814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:02.653721 5817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:02.655215 5807 server_base.cc:1034] running on GCE node
I20260928 16:42:02.655932 5807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:02.656848 5807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:02.658161 5807 hybrid_clock.cc:648] HybridClock initialized: now 1790613722658117 us; error 77 us; skew 500 ppm
Sep 28 16:42:02 dist-test-slave-nqgh krb5kdc[5250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613722, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:02.661868 5807 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:42:02.663321 5807 webserver.cc:460] Webserver started at http://127.25.71.65:43607/ using document root <none> and password file <none>
I20260928 16:42:02.664331 5807 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:02.664518 5807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:02.664875 5807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:42:02 dist-test-slave-nqgh krb5kdc[5250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613722, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
I20260928 16:42:02.693064 5807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "99fa3182455a4c398a3563d5fc49c237"
format_stamp: "Formatted at 2026-09-28 16:42:02 on dist-test-slave-nqgh"
server_key: "c0887ed3293ca75f7b0ae7682b93edb7"
server_key_iv: "b72657eb44e22fca844501e78c361a2a"
server_key_version: "kuduclusterkey@0"
I20260928 16:42:02.693949 5807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "99fa3182455a4c398a3563d5fc49c237"
format_stamp: "Formatted at 2026-09-28 16:42:02 on dist-test-slave-nqgh"
server_key: "c0887ed3293ca75f7b0ae7682b93edb7"
server_key_iv: "b72657eb44e22fca844501e78c361a2a"
server_key_version: "kuduclusterkey@0"
I20260928 16:42:02.703488 5807 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20260928 16:42:02.726253 5826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:02.728127 5807 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260928 16:42:02.728286 5807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "99fa3182455a4c398a3563d5fc49c237"
format_stamp: "Formatted at 2026-09-28 16:42:02 on dist-test-slave-nqgh"
server_key: "c0887ed3293ca75f7b0ae7682b93edb7"
server_key_iv: "b72657eb44e22fca844501e78c361a2a"
server_key_version: "kuduclusterkey@0"
I20260928 16:42:02.728436 5807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:02.755264 5807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:02.758323 5807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:02.758510 5807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:02.759148 5807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:42:02.760076 5807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:42:02.760125 5807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:02.760179 5807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:42:02.760226 5807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:02.772168 5807 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:38683
I20260928 16:42:02.772202 5939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:38683 every 8 connection(s)
I20260928 16:42:02.773406 5807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:42:02.781042 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 5807
I20260928 16:42:02.781220 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:42:02.781622 25885 external_mini_cluster.cc:1428] Setting key eaa254f903168d755120cd4201b9c79d
Sep 28 16:42:02 dist-test-slave-nqgh krb5kdc[5250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613722, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:02.789671 5942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:02.775534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:42215 (local address 127.25.71.126:35713)
0928 16:42:02.778824 (+ 3290us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:02.778828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:02.778842 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:42:02.778886 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:02.778888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:02.778943 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:02.779029 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:02.780275 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:02.780862 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:02.782111 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:02.782407 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:02.782495 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:02.785288 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:02.785291 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:02.785294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:02.785333 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:02.787263 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:02.787916 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:02.787920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:02.787921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:02.788025 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:02.788383 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:02.788386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:02.788387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:02.788663 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:02.788767 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:02.789323 (+ 556us) server_negotiation.cc:299] Negotiation successful
0928 16:42:02.789478 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3200,"thread_start_us":103,"threads_started":1}
I20260928 16:42:02.790666 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:02.775932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35713 (local address 127.25.71.65:42215)
0928 16:42:02.776459 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:02.776515 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:02.777454 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:02.779257 (+ 1803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:02.779265 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:02.779571 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:02.780110 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:02.780127 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:02.780996 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:02.780999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:02.781981 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:02.781993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:02.782761 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:02.782765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:02.782934 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:02.783548 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:02.783570 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:02.785095 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:02.787410 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:02.787433 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:02.787451 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:02.787788 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:02.788119 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:02.788122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:02.788124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:02.788282 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:02.788768 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:02.788776 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:02.789074 (+ 298us) client_negotiation.cc:769] Sending connection context
0928 16:42:02.789261 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 16:42:02.789628 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":124,"threads_started":1}
I20260928 16:42:02.792204 5940 heartbeater.cc:347] Connected to a master server at 127.25.71.126:35713
I20260928 16:42:02.792461 5940 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:02.792999 5940 heartbeater.cc:511] Master 127.25.71.126:35713 requested a full tablet report, sending...
I20260928 16:42:02.794608 5721 ts_manager.cc:194] Registered new tserver with Master: 99fa3182455a4c398a3563d5fc49c237 (127.25.71.65:38683)
I20260928 16:42:02.796193 5721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:42215
WARNING: no policy specified for kudu/127.25.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.66@KRBTEST.COM" created.
Sep 28 16:42:02 dist-test-slave-nqgh krb5kdc[5250](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790613722, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
I20260928 16:42:02.832137 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.71.84:45447/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:35713
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:42:03.032716 5946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:03.033347 5946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:03.033427 5946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:03.040724 5946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:03.040940 5946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:03.041096 5946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:42:03.051576 5946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.25.71.84:45447/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:35713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:03.053602 5946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:03.055693 5946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:03.072484 5953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:03.072577 5952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:03.072669 5955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:03.073331 5946 server_base.cc:1034] running on GCE node
I20260928 16:42:03.074049 5946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:03.075286 5946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:03.076541 5946 hybrid_clock.cc:648] HybridClock initialized: now 1790613723076492 us; error 63 us; skew 500 ppm
Sep 28 16:42:03 dist-test-slave-nqgh krb5kdc[5250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613723, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:03.081408 5946 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:42:03.083212 5946 webserver.cc:460] Webserver started at http://127.25.71.66:35913/ using document root <none> and password file <none>
I20260928 16:42:03.084245 5946 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:03.084379 5946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:03.084738 5946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:42:03 dist-test-slave-nqgh krb5kdc[5250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613723, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
I20260928 16:42:03.111557 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "173e35deb6ac4e69bacc5bbc60211340"
format_stamp: "Formatted at 2026-09-28 16:42:03 on dist-test-slave-nqgh"
server_key: "b427d0086cdb4d95eae4ee001cf056ba"
server_key_iv: "d35221e5cfcec4a90a3b0726e357bbc4"
server_key_version: "kuduclusterkey@0"
I20260928 16:42:03.112869 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "173e35deb6ac4e69bacc5bbc60211340"
format_stamp: "Formatted at 2026-09-28 16:42:03 on dist-test-slave-nqgh"
server_key: "b427d0086cdb4d95eae4ee001cf056ba"
server_key_iv: "d35221e5cfcec4a90a3b0726e357bbc4"
server_key_version: "kuduclusterkey@0"
I20260928 16:42:03.120333 5946 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260928 16:42:03.140749 5962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:03.142551 5946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:42:03.142750 5946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "173e35deb6ac4e69bacc5bbc60211340"
format_stamp: "Formatted at 2026-09-28 16:42:03 on dist-test-slave-nqgh"
server_key: "b427d0086cdb4d95eae4ee001cf056ba"
server_key_iv: "d35221e5cfcec4a90a3b0726e357bbc4"
server_key_version: "kuduclusterkey@0"
I20260928 16:42:03.142927 5946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:03.157230 5946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:03.160481 5946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:03.160662 5946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:03.161433 5946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:42:03.162519 5946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:42:03.162582 5946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:03.162665 5946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:42:03.162701 5946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:03.173306 5946 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:38987
I20260928 16:42:03.173422 6075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:38987 every 8 connection(s)
I20260928 16:42:03.174376 5946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:42:03.179005 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 5946
I20260928 16:42:03.179162 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:42:03.179600 25885 external_mini_cluster.cc:1428] Setting key 9e0dfa2246f167bfc0cec42a36da7c90
WARNING: no policy specified for kudu/127.25.71.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.67@KRBTEST.COM" created.
Sep 28 16:42:03 dist-test-slave-nqgh krb5kdc[5250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613723, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:03.224874 5942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:03.199855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:47525 (local address 127.25.71.126:35713)
0928 16:42:03.200048 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:03.200054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:03.201613 (+ 1559us) server_negotiation.cc:407] Connection header received
0928 16:42:03.203091 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:03.203096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:03.203163 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:03.203216 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:03.205953 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:03.206918 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:03.209832 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:03.210335 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:03.210461 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:03.219334 (+ 8873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:03.219339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:03.219342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:03.219385 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:03.221496 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:03.222405 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:03.222409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:03.222411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:03.222515 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:03.223068 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:03.223072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:03.223073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:03.223376 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:03.223458 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:03.224462 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0928 16:42:03.224647 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 16:42:03.227136 6080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:03.199858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35713 (local address 127.25.71.66:47525)
0928 16:42:03.201400 (+ 1542us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:03.201470 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:03.202794 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:03.203700 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:03.203710 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:03.204114 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:03.204852 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:03.204874 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:03.207262 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:03.207267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:03.209518 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:03.209553 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:03.210871 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:03.210877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:03.211196 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:03.212138 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:03.212168 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:03.218888 (+ 6720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:03.221704 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:03.221714 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:03.221740 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:03.222209 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:03.222671 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:03.222674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:03.222677 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:03.222938 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:03.223552 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:03.223561 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:03.224105 (+ 544us) client_negotiation.cc:769] Sending connection context
0928 16:42:03.225175 (+ 1070us) client_negotiation.cc:240] Negotiation successful
0928 16:42:03.225754 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1270,"thread_start_us":227,"threads_started":1}
I20260928 16:42:03.228816 6077 heartbeater.cc:347] Connected to a master server at 127.25.71.126:35713
I20260928 16:42:03.229357 6077 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
I20260928 16:42:03.230394 6077 heartbeater.cc:511] Master 127.25.71.126:35713 requested a full tablet report, sending...
I20260928 16:42:03.230540 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.71.84:45447/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:35713
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:42:03.232470 5721 ts_manager.cc:194] Registered new tserver with Master: 173e35deb6ac4e69bacc5bbc60211340 (127.25.71.66:38987)
I20260928 16:42:03.233357 5721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:47525
Sep 28 16:42:03 dist-test-slave-nqgh krb5kdc[5250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613722, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
W20260928 16:42:03.414057 6081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:03.414535 6081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:03.414700 6081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:03.421806 6081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:03.422043 6081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:03.422183 6081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:42:03.431222 6081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.25.71.84:45447/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:35713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:03.433003 6081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:03.434618 6081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:03.450052 6088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:03.450050 6086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:03.451092 6090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:03.451804 6081 server_base.cc:1034] running on GCE node
I20260928 16:42:03.452562 6081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:03.453953 6081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:03.457284 6081 hybrid_clock.cc:648] HybridClock initialized: now 1790613723456729 us; error 568 us; skew 500 ppm
Sep 28 16:42:03 dist-test-slave-nqgh krb5kdc[5250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613723, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:03.461753 6081 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:42:03.463469 6081 webserver.cc:460] Webserver started at http://127.25.71.67:39845/ using document root <none> and password file <none>
I20260928 16:42:03.464419 6081 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:03.464540 6081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:03.464908 6081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:42:03 dist-test-slave-nqgh krb5kdc[5250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613723, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
I20260928 16:42:03.487030 6081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "0431291fa0624d708fe8d5f41049cc41"
format_stamp: "Formatted at 2026-09-28 16:42:03 on dist-test-slave-nqgh"
server_key: "d8c547a4f8d83db7f8524b17ec18286d"
server_key_iv: "263009ff245dbead9dd560634acaffeb"
server_key_version: "kuduclusterkey@0"
I20260928 16:42:03.489212 6081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "0431291fa0624d708fe8d5f41049cc41"
format_stamp: "Formatted at 2026-09-28 16:42:03 on dist-test-slave-nqgh"
server_key: "d8c547a4f8d83db7f8524b17ec18286d"
server_key_iv: "263009ff245dbead9dd560634acaffeb"
server_key_version: "kuduclusterkey@0"
I20260928 16:42:03.494951 6081 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:42:03.527355 6097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:03.529942 6081 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.004s sys 0.000s
I20260928 16:42:03.530166 6081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "0431291fa0624d708fe8d5f41049cc41"
format_stamp: "Formatted at 2026-09-28 16:42:03 on dist-test-slave-nqgh"
server_key: "d8c547a4f8d83db7f8524b17ec18286d"
server_key_iv: "263009ff245dbead9dd560634acaffeb"
server_key_version: "kuduclusterkey@0"
I20260928 16:42:03.530360 6081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:03.567171 6081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:03.571764 6081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:03.572017 6081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:03.572947 6081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:42:03.574371 6081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:42:03.574456 6081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:03.574617 6081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:42:03.574688 6081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:03.589443 6081 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:35561
I20260928 16:42:03.589818 6210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:35561 every 8 connection(s)
I20260928 16:42:03.591177 6081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:42:03.593353 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 6081
I20260928 16:42:03.593488 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:42:03.593843 25885 external_mini_cluster.cc:1428] Setting key f2ef6d8ed2f2179dd278613dc6320247
Sep 28 16:42:03 dist-test-slave-nqgh krb5kdc[5250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613723, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:03.611421 5942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:03.594667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:43011 (local address 127.25.71.126:35713)
0928 16:42:03.594849 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:03.594855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:03.596355 (+ 1500us) server_negotiation.cc:407] Connection header received
0928 16:42:03.597790 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:03.597794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:03.597863 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:03.597948 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:03.599873 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:03.600774 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:03.602539 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:03.603013 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:03.603145 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:03.606792 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:03.606797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:03.606801 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:03.606841 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:03.608692 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:03.609457 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:03.609461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:03.609463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:03.609553 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:03.609974 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:03.609978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:03.609980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:03.610226 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:03.610301 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:03.611099 (+ 798us) server_negotiation.cc:299] Negotiation successful
0928 16:42:03.611246 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:42:03.613372 6213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:03.594315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35713 (local address 127.25.71.67:43011)
0928 16:42:03.596107 (+ 1792us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:03.596178 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:03.597317 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:03.598501 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:03.598511 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:03.598951 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:03.599674 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:03.599694 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:03.600950 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:03.600954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:03.602359 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:03.602372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:03.603262 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:03.603266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:03.603488 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:03.604306 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:03.604327 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:03.606431 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:03.608894 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:03.608902 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:03.608925 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:03.609329 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:03.609653 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:03.609656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:03.609659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:03.609860 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:03.610370 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:03.610377 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:03.610798 (+ 421us) client_negotiation.cc:769] Sending connection context
0928 16:42:03.611904 (+ 1106us) client_negotiation.cc:240] Negotiation successful
0928 16:42:03.612287 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1515,"thread_start_us":208,"threads_started":1}
I20260928 16:42:03.614952 6211 heartbeater.cc:347] Connected to a master server at 127.25.71.126:35713
I20260928 16:42:03.615226 6211 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:03.615895 6211 heartbeater.cc:511] Master 127.25.71.126:35713 requested a full tablet report, sending...
I20260928 16:42:03.617313 5721 ts_manager.cc:194] Registered new tserver with Master: 0431291fa0624d708fe8d5f41049cc41 (127.25.71.67:35561)
I20260928 16:42:03.618114 5721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:43011
I20260928 16:42:03.626824 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:42:03.627970 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 5807
I20260928 16:42:03.634827 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 5946
I20260928 16:42:03.644333 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 6081
I20260928 16:42:03.653676 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 5694
I20260928 16:42:03.663578 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:35713
--webserver_interface=127.25.71.126
--webserver_port=39619
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:35713
--ranger_config_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.71.84:45447/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.71.84:45447/kms with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:42:03.806453 6214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:03.806923 6214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:03.806977 6214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:03.811615 6214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:42:03.811712 6214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:03.811731 6214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:03.811748 6214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:42:03.811762 6214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:42:03.818454 6214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.71.84:45447/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:35713
--ranger_config_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:35713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=39619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:03.819998 6214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:03.821322 6214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:03.832471 6222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:03.832473 6220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:03.832471 6219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:03.833007 6214 server_base.cc:1034] running on GCE node
I20260928 16:42:03.833734 6214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:03.835115 6214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:03.836346 6214 hybrid_clock.cc:648] HybridClock initialized: now 1790613723836302 us; error 59 us; skew 500 ppm
Sep 28 16:42:03 dist-test-slave-nqgh krb5kdc[5250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613723, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:03.839830 6214 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:42:03.841158 6214 webserver.cc:460] Webserver started at http://127.25.71.126:39619/ using document root <none> and password file <none>
I20260928 16:42:03.841977 6214 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:03.842073 6214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 16:42:03 dist-test-slave-nqgh krb5kdc[5250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613723, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
I20260928 16:42:03.899816 6214 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:42:03.903709 6230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:03.905884 6214 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:42:03.906131 6214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "40d30c200c4b4935bfa2bdedd1edf9a2"
format_stamp: "Formatted at 2026-09-28 16:42:01 on dist-test-slave-nqgh"
server_key: "d1461ec2f29eb8e8cbe40463dc5fd529"
server_key_iv: "300b3cef02bb76d2b8a6dd7b7f29d24f"
server_key_version: "kuduclusterkey@0"
I20260928 16:42:03.906970 6214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:03.929672 6214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:03.933311 6214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:03.933555 6214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:03.942682 6214 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:35713
I20260928 16:42:03.942708 6282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:35713 every 8 connection(s)
I20260928 16:42:03.944278 6214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:42:03.949579 6283 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:42:03.951642 6283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2: Bootstrap starting.
I20260928 16:42:03.954146 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 6214
I20260928 16:42:03.954713 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.71.84:45447/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:38683
--local_ip_for_outbound_sockets=127.25.71.65
--tserver_master_addrs=127.25.71.126:35713
--webserver_port=43607
--webserver_interface=127.25.71.65
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.71.84:45447/kms with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:42:03.968971 6283 log.cc:824] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:03.976366 6283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:42:03.976819 6283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2: Bootstrap complete.
I20260928 16:42:03.981379 6283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d30c200c4b4935bfa2bdedd1edf9a2" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35713 } }
I20260928 16:42:03.981855 6283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d30c200c4b4935bfa2bdedd1edf9a2, State: Initialized, Role: FOLLOWER
I20260928 16:42:03.982424 6283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d30c200c4b4935bfa2bdedd1edf9a2" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35713 } }
I20260928 16:42:03.982563 6283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:42:03.982661 6283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:42:03.982782 6283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:42:03.985083 6283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d30c200c4b4935bfa2bdedd1edf9a2" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35713 } }
I20260928 16:42:03.985483 6283 leader_election.cc:302] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40d30c200c4b4935bfa2bdedd1edf9a2; no voters:
I20260928 16:42:03.985834 6283 leader_election.cc:288] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 16:42:03.985951 6287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:42:03.986286 6287 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [term 2 LEADER]: Becoming Leader. State: Replica: 40d30c200c4b4935bfa2bdedd1edf9a2, State: Running, Role: LEADER
I20260928 16:42:03.986702 6287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d30c200c4b4935bfa2bdedd1edf9a2" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35713 } }
I20260928 16:42:03.987380 6283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:42:03.988197 6288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "40d30c200c4b4935bfa2bdedd1edf9a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d30c200c4b4935bfa2bdedd1edf9a2" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35713 } } }
I20260928 16:42:03.988363 6288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [sys.catalog]: This master's current role is: LEADER
I20260928 16:42:03.988695 6289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40d30c200c4b4935bfa2bdedd1edf9a2. Latest consensus state: current_term: 2 leader_uuid: "40d30c200c4b4935bfa2bdedd1edf9a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d30c200c4b4935bfa2bdedd1edf9a2" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35713 } } }
I20260928 16:42:03.988809 6289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2 [sys.catalog]: This master's current role is: LEADER
I20260928 16:42:03.989804 6295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:42:03.990342 6283 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 16:42:03.998232 6295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:42:03.999506 6295 catalog_manager.cc:1259] Loaded cluster ID: 6f7f66d4dea64769b456e2881edc8e0a
I20260928 16:42:03.999614 6295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:42:04.007040 6295 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:42:04.008378 6295 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2: Loaded TSK: 0
I20260928 16:42:04.010252 6295 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:42:04.103559 6285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:04.103963 6285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:04.104034 6285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:04.110888 6285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:04.111019 6285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:04.111129 6285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:42:04.119421 6285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.71.84:45447/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:38683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=43607
--tserver_master_addrs=127.25.71.126:35713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:04.120965 6285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:04.122364 6285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:04.137279 6323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:04.137619 6321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:04.137921 6320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:04.138394 6285 server_base.cc:1034] running on GCE node
I20260928 16:42:04.139190 6285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:04.140110 6285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:04.143481 6285 hybrid_clock.cc:648] HybridClock initialized: now 1790613724143425 us; error 91 us; skew 500 ppm
Sep 28 16:42:04 dist-test-slave-nqgh krb5kdc[5250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613724, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:04.148466 6285 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:42:04.150373 6285 webserver.cc:460] Webserver started at http://127.25.71.65:43607/ using document root <none> and password file <none>
I20260928 16:42:04.151340 6285 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:04.151501 6285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 16:42:04 dist-test-slave-nqgh krb5kdc[5250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613724, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
I20260928 16:42:04.185058 6285 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:42:04.192760 6331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:04.194969 6285 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260928 16:42:04.195154 6285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "99fa3182455a4c398a3563d5fc49c237"
format_stamp: "Formatted at 2026-09-28 16:42:02 on dist-test-slave-nqgh"
server_key: "c0887ed3293ca75f7b0ae7682b93edb7"
server_key_iv: "b72657eb44e22fca844501e78c361a2a"
server_key_version: "kuduclusterkey@0"
I20260928 16:42:04.195999 6285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:04.240378 6285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:04.244621 6285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:04.244853 6285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:04.245689 6285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:42:04.247498 6285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:42:04.247664 6285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:04.247762 6285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:42:04.247795 6285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:04.278403 6285 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:38683
I20260928 16:42:04.278678 6444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:38683 every 8 connection(s)
I20260928 16:42:04.279918 6285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:42:04.288559 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 6285
I20260928 16:42:04.289109 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.71.84:45447/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:38987
--local_ip_for_outbound_sockets=127.25.71.66
--tserver_master_addrs=127.25.71.126:35713
--webserver_port=35913
--webserver_interface=127.25.71.66
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.71.84:45447/kms with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:42:04 dist-test-slave-nqgh krb5kdc[5250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613724, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:04.303709 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:04.283055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35713 (local address 127.25.71.65:50003)
0928 16:42:04.283728 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:04.283792 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:04.284818 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:04.286511 (+ 1693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:04.286520 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:04.286960 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:04.287513 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:04.287530 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:04.289689 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:04.289694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:04.291309 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:04.291327 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:04.292886 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:04.292890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:04.293118 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:04.293995 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:04.294021 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:04.296108 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:04.299293 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:04.299302 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:04.299326 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:04.299777 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:04.300271 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:04.300275 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:04.300277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:04.300453 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:04.301438 (+ 985us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:04.301445 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:04.301801 (+ 356us) client_negotiation.cc:769] Sending connection context
0928 16:42:04.302054 (+ 253us) client_negotiation.cc:240] Negotiation successful
0928 16:42:04.302528 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":166,"threads_started":1}
I20260928 16:42:04.306141 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:04.283045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:50003 (local address 127.25.71.126:35713)
0928 16:42:04.285349 (+ 2304us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:04.285370 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:04.285411 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:42:04.285691 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:04.285702 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:04.286062 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:04.286401 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:04.287763 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:04.289467 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:04.291501 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:04.292374 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:04.292586 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:04.296551 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:04.296562 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:04.296595 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:04.296659 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:04.299070 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:04.299952 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:04.299961 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:04.299967 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:04.300155 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:04.300576 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:04.300580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:04.300581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:04.301213 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:04.301382 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:04.304810 (+ 3428us) server_negotiation.cc:299] Negotiation successful
0928 16:42:04.305181 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2104,"thread_start_us":170,"threads_started":1}
I20260928 16:42:04.309672 6445 heartbeater.cc:347] Connected to a master server at 127.25.71.126:35713
I20260928 16:42:04.310072 6445 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:04.311126 6445 heartbeater.cc:511] Master 127.25.71.126:35713 requested a full tablet report, sending...
W20260928 16:42:04.314483 6445 heartbeater.cc:644] Failed to heartbeat to 127.25.71.126:35713 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 16:42:04.314725 6445 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:04.315006 6445 heartbeater.cc:511] Master 127.25.71.126:35713 requested a full tablet report, sending...
I20260928 16:42:04.316131 6445 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:04.316349 6445 heartbeater.cc:511] Master 127.25.71.126:35713 requested a full tablet report, sending...
W20260928 16:42:04.317166 6445 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:42:04.477854 6449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:04.478291 6449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:04.478463 6449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:04.485143 6449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:04.485272 6449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:04.485373 6449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:42:04.494277 6449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.71.84:45447/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:38987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=35913
--tserver_master_addrs=127.25.71.126:35713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:04.496114 6449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:04.497673 6449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:04.513258 6455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:04.513278 6457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:04.514603 6449 server_base.cc:1034] running on GCE node
W20260928 16:42:04.514618 6454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:04.515479 6449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:04.516383 6449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:04.521811 6449 hybrid_clock.cc:648] HybridClock initialized: now 1790613724521732 us; error 66 us; skew 500 ppm
Sep 28 16:42:04 dist-test-slave-nqgh krb5kdc[5250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613724, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:04.527323 6449 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:42:04.529141 6449 webserver.cc:460] Webserver started at http://127.25.71.66:35913/ using document root <none> and password file <none>
I20260928 16:42:04.530218 6449 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:04.530445 6449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 16:42:04 dist-test-slave-nqgh krb5kdc[5250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613724, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
I20260928 16:42:04.556968 6449 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 16:42:04.561399 6466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:04.563674 6449 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260928 16:42:04.563874 6449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "173e35deb6ac4e69bacc5bbc60211340"
format_stamp: "Formatted at 2026-09-28 16:42:03 on dist-test-slave-nqgh"
server_key: "b427d0086cdb4d95eae4ee001cf056ba"
server_key_iv: "d35221e5cfcec4a90a3b0726e357bbc4"
server_key_version: "kuduclusterkey@0"
I20260928 16:42:04.564859 6449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:04.597044 6449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:04.602226 6449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:04.602458 6449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:04.603335 6449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:42:04.604487 6449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:42:04.604588 6449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:04.604662 6449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:42:04.604692 6449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:04.617357 6449 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:38987
I20260928 16:42:04.617535 6579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:38987 every 8 connection(s)
I20260928 16:42:04.618891 6449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:42:04.623106 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 6449
I20260928 16:42:04.623668 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.71.84:45447/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:35561
--local_ip_for_outbound_sockets=127.25.71.67
--tserver_master_addrs=127.25.71.126:35713
--webserver_port=39845
--webserver_interface=127.25.71.67
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.71.84:45447/kms with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:42:04 dist-test-slave-nqgh krb5kdc[5250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613724, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:04.644339 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:04.621797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:60719 (local address 127.25.71.126:35713)
0928 16:42:04.621967 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:04.621972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:04.628272 (+ 6300us) server_negotiation.cc:407] Connection header received
0928 16:42:04.629866 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:04.629871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:04.629972 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:04.630049 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:04.632195 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:04.633109 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:04.635292 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:04.635698 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:04.635870 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:04.638982 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:04.638986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:04.638989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:04.639025 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:04.640903 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:04.642147 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:04.642153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:04.642155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:04.642289 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:04.642799 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:04.642803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:04.642805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:04.643078 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:04.643165 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:04.643970 (+ 805us) server_negotiation.cc:299] Negotiation successful
0928 16:42:04.644153 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 16:42:04.646605 6582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:04.622349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35713 (local address 127.25.71.66:60719)
0928 16:42:04.627914 (+ 5565us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:04.628052 (+ 138us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:04.629450 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:04.630724 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:04.630738 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:04.631188 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:04.631914 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:04.631935 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:04.633335 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:04.633340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:04.635069 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:04.635092 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:04.636050 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:04.636057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:04.636323 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:04.637047 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:04.637092 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:04.638708 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:04.641232 (+ 2524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:04.641244 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:04.641269 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:04.641958 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:04.642411 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:04.642415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:04.642417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:04.642620 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:04.643236 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:04.643244 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:04.643652 (+ 408us) client_negotiation.cc:769] Sending connection context
0928 16:42:04.644875 (+ 1223us) client_negotiation.cc:240] Negotiation successful
0928 16:42:04.645448 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5144,"thread_start_us":192,"threads_started":1}
I20260928 16:42:04.649049 6580 heartbeater.cc:347] Connected to a master server at 127.25.71.126:35713
I20260928 16:42:04.649521 6580 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:04.650484 6580 heartbeater.cc:511] Master 127.25.71.126:35713 requested a full tablet report, sending...
W20260928 16:42:04.653293 6580 heartbeater.cc:644] Failed to heartbeat to 127.25.71.126:35713 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 16:42:04.653476 6580 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:04.653733 6580 heartbeater.cc:511] Master 127.25.71.126:35713 requested a full tablet report, sending...
I20260928 16:42:04.654814 6580 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:04.655220 6580 heartbeater.cc:511] Master 127.25.71.126:35713 requested a full tablet report, sending...
W20260928 16:42:04.656132 6580 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:42:04.795531 6583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:04.795866 6583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:04.795939 6583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:04.800372 6583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:04.800475 6583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:04.800565 6583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:42:04.807274 6583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.71.84:45447/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:35561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=39845
--tserver_master_addrs=127.25.71.126:35713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:04.808966 6583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:04.810422 6583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:04.822261 6589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:04.822348 6588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:04.822302 6591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:04.822588 6583 server_base.cc:1034] running on GCE node
I20260928 16:42:04.823521 6583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:04.824483 6583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:04.827813 6583 hybrid_clock.cc:648] HybridClock initialized: now 1790613724827760 us; error 53 us; skew 500 ppm
Sep 28 16:42:04 dist-test-slave-nqgh krb5kdc[5250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613724, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:04.831985 6583 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:42:04.833165 6583 webserver.cc:460] Webserver started at http://127.25.71.67:39845/ using document root <none> and password file <none>
I20260928 16:42:04.833877 6583 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:04.834065 6583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 16:42:04 dist-test-slave-nqgh krb5kdc[5250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613724, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
I20260928 16:42:04.877674 6583 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:42:04.887655 6599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:04.890519 6583 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.000s sys 0.002s
I20260928 16:42:04.890919 6583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "0431291fa0624d708fe8d5f41049cc41"
format_stamp: "Formatted at 2026-09-28 16:42:03 on dist-test-slave-nqgh"
server_key: "d8c547a4f8d83db7f8524b17ec18286d"
server_key_iv: "263009ff245dbead9dd560634acaffeb"
server_key_version: "kuduclusterkey@0"
I20260928 16:42:04.891942 6583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:04.920338 6583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:04.924324 6583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:04.924623 6583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:04.925498 6583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:42:04.926764 6583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:42:04.926872 6583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:04.926975 6583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:42:04.927039 6583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:04.940963 6583 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:35561
I20260928 16:42:04.941273 6719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:35561 every 8 connection(s)
I20260928 16:42:04.942607 6583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:42:04.943264 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 6583
Sep 28 16:42:04 dist-test-slave-nqgh krb5kdc[5250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613724, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:04.958755 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:04.944888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32996 (local address 127.25.71.126:35713)
0928 16:42:04.945045 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:04.945052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:04.946438 (+ 1386us) server_negotiation.cc:407] Connection header received
0928 16:42:04.946801 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:04.946805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:04.946870 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:04.946925 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:04.948061 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:04.948853 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:04.950327 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:04.950730 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:04.950804 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:04.952870 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:04.952874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:04.952877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:04.952913 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:04.954769 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:04.956758 (+ 1989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:04.956763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:04.956766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:04.956892 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:04.957293 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:04.957297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:04.957299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:04.957591 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:04.957826 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:04.958329 (+ 503us) server_negotiation.cc:299] Negotiation successful
0928 16:42:04.958518 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:42:04.966260 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:04.947928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:48911 (local address 127.25.71.126:35713)
0928 16:42:04.950159 (+ 2231us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:04.950168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:04.950184 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:42:04.950237 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:04.950240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:04.950299 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:04.950384 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:04.952264 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:04.953175 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:04.955079 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:04.955447 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:04.955535 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:04.960695 (+ 5160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:04.960699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:04.960702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:04.960733 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:04.962866 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:04.963837 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:04.963841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:04.963844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:04.963980 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:04.964536 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:04.964539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:04.964541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:04.964830 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:04.964953 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:04.965867 (+ 914us) server_negotiation.cc:299] Negotiation successful
0928 16:42:04.966055 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2132,"thread_start_us":121,"threads_started":1}
I20260928 16:42:04.967777 6723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:04.946610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35713 (local address 127.25.71.67:48911)
0928 16:42:04.948462 (+ 1852us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:04.948543 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:04.949667 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:04.950941 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:04.950964 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:04.951349 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:04.952068 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:04.952089 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:04.953367 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:04.953371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:04.954924 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:04.954938 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:04.955807 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:04.955810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:04.956057 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:04.956797 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:04.956820 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:04.959814 (+ 2994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:04.963083 (+ 3269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:04.963095 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:04.963123 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:04.963652 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:04.964107 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:04.964110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:04.964113 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:04.964383 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:04.964963 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:04.964972 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:04.965481 (+ 509us) client_negotiation.cc:769] Sending connection context
0928 16:42:04.965754 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 16:42:04.966272 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1543,"thread_start_us":177,"threads_started":1}
I20260928 16:42:04.969990 6720 heartbeater.cc:347] Connected to a master server at 127.25.71.126:35713
I20260928 16:42:04.970322 6720 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:04.971117 6720 heartbeater.cc:511] Master 127.25.71.126:35713 requested a full tablet report, sending...
I20260928 16:42:04.972930 6247 ts_manager.cc:194] Registered new tserver with Master: 0431291fa0624d708fe8d5f41049cc41 (127.25.71.67:35561)
I20260928 16:42:04.974355 6247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:48911
I20260928 16:42:05.317477 6445 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:05.317888 6445 heartbeater.cc:511] Master 127.25.71.126:35713 requested a full tablet report, sending...
I20260928 16:42:05.319175 6247 ts_manager.cc:194] Registered new tserver with Master: 99fa3182455a4c398a3563d5fc49c237 (127.25.71.65:38683)
I20260928 16:42:05.320029 6247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:50003
Sep 28 16:42:05 dist-test-slave-nqgh krb5kdc[5250](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790613725, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:05.656457 6580 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:05.656909 6580 heartbeater.cc:511] Master 127.25.71.126:35713 requested a full tablet report, sending...
I20260928 16:42:05.657930 6247 ts_manager.cc:194] Registered new tserver with Master: 173e35deb6ac4e69bacc5bbc60211340 (127.25.71.66:38987)
I20260928 16:42:05.658788 6247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:60719
I20260928 16:42:05.658913 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:42:05.687804 6735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:05.668225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33004 (local address 127.25.71.126:35713)
0928 16:42:05.670077 (+ 1852us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:05.670081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:05.670109 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:42:05.670166 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:05.670169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:05.670227 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:05.670335 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:05.671421 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.672261 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.673539 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.673876 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.673999 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:05.681893 (+ 7894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:05.681899 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:05.681903 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:05.681964 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:05.684370 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:05.685219 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:05.685225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:05.685228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:05.685388 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:05.685920 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:05.685923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:05.685926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:05.686214 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:05.686388 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:05.687380 (+ 992us) server_negotiation.cc:299] Negotiation successful
0928 16:42:05.687570 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1743,"thread_start_us":118,"threads_started":1}
I20260928 16:42:05.694136 6247 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:42:05.700477 6247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:42:05.732501 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:05.720072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:35561 (local address 127.0.0.1:54340)
0928 16:42:05.722932 (+ 2860us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:05.722945 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:05.723112 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:05.724963 (+ 1851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:05.724966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:05.724986 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:05.725130 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:05.725137 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.726650 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.726655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:05.729149 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:05.729163 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.730744 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.730747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:05.730951 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:05.732019 (+ 1068us) client_negotiation.cc:769] Sending connection context
0928 16:42:05.732152 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 16:42:05.732317 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2792,"mutex_wait_us":42,"thread_start_us":104,"threads_started":1}
I20260928 16:42:05.733177 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:05.720418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54340 (local address 127.25.71.67:35561)
0928 16:42:05.721765 (+ 1347us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:05.721774 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:05.723032 (+ 1258us) server_negotiation.cc:407] Connection header received
0928 16:42:05.723280 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:05.723287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:05.724565 (+ 1278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:05.724723 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:05.725270 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.726479 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.729322 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.730290 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.730386 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:05.730582 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:05.732905 (+ 2323us) server_negotiation.cc:299] Negotiation successful
0928 16:42:05.733040 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1217,"thread_start_us":100,"threads_started":1}
I20260928 16:42:05.733103 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:05.719288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:38987 (local address 127.0.0.1:47916)
0928 16:42:05.722573 (+ 3285us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:05.722585 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:05.722723 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:05.724696 (+ 1973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:05.724699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:05.724723 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:05.724836 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:05.724844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.726427 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.726432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:05.730903 (+ 4471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:05.730916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.732568 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.732572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:05.732693 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:05.732723 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 16:42:05.732801 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 16:42:05.732903 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3224,"mutex_wait_us":2886,"thread_start_us":113,"threads_started":1}
I20260928 16:42:05.735692 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:05.719517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47916 (local address 127.25.71.66:38987)
0928 16:42:05.721263 (+ 1746us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:05.721274 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:05.723913 (+ 2639us) server_negotiation.cc:407] Connection header received
0928 16:42:05.724026 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:05.724033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:05.724272 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:05.724455 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:05.724980 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.726262 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.731090 (+ 4828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.732150 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.732279 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:05.732488 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:05.735243 (+ 2755us) server_negotiation.cc:299] Negotiation successful
0928 16:42:05.735432 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1578,"thread_start_us":148,"threads_started":1}
I20260928 16:42:05.737138 6744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:05.717902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:38683 (local address 127.0.0.1:34908)
0928 16:42:05.722070 (+ 4168us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:05.722143 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:05.722309 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:05.723484 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:05.723492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:05.723528 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:05.723734 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:05.723743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.728642 (+ 4899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.728646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:05.729820 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:05.729835 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.736533 (+ 6698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.736539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:05.736670 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:05.736707 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 16:42:05.736801 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 16:42:05.736942 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3951,"thread_start_us":134,"threads_started":1}
I20260928 16:42:05.737279 6654 tablet_service.cc:1467] Processing CreateTablet for tablet b9a8ba1cbb074d4c9ae645a80e0adc56 (DEFAULT_TABLE table=test-table [id=931f3b46300445abb81161de59ef9db8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:42:05.738618 6654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9a8ba1cbb074d4c9ae645a80e0adc56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:05.741911 6514 tablet_service.cc:1467] Processing CreateTablet for tablet b9a8ba1cbb074d4c9ae645a80e0adc56 (DEFAULT_TABLE table=test-table [id=931f3b46300445abb81161de59ef9db8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:42:05.745200 6514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9a8ba1cbb074d4c9ae645a80e0adc56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:05.750795 6746 tablet_bootstrap.cc:492] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 0431291fa0624d708fe8d5f41049cc41: Bootstrap starting.
I20260928 16:42:05.752216 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:05.718197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.25.71.65:38683)
0928 16:42:05.721517 (+ 3320us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:05.721524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:05.722299 (+ 775us) server_negotiation.cc:407] Connection header received
0928 16:42:05.722520 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:05.722528 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:05.722931 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:05.723192 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:05.726918 (+ 3726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.728196 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.733290 (+ 5094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.736319 (+ 3029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.751379 (+ 15060us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:05.751692 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:05.751812 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 16:42:05.751987 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3189,"thread_start_us":133,"threads_started":1}
I20260928 16:42:05.753538 6746 tablet_bootstrap.cc:654] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 0431291fa0624d708fe8d5f41049cc41: Neither blocks nor log segments found. Creating new log.
I20260928 16:42:05.754617 6746 log.cc:824] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 0431291fa0624d708fe8d5f41049cc41: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:05.757321 6746 tablet_bootstrap.cc:492] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 0431291fa0624d708fe8d5f41049cc41: No bootstrap required, opened a new log
I20260928 16:42:05.757547 6746 ts_tablet_manager.cc:1397] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 0431291fa0624d708fe8d5f41049cc41: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:42:05.759575 6378 tablet_service.cc:1467] Processing CreateTablet for tablet b9a8ba1cbb074d4c9ae645a80e0adc56 (DEFAULT_TABLE table=test-table [id=931f3b46300445abb81161de59ef9db8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:42:05.762908 6746 raft_consensus.cc:348] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 0431291fa0624d708fe8d5f41049cc41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fa3182455a4c398a3563d5fc49c237" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 38683 } } peers { permanent_uuid: "173e35deb6ac4e69bacc5bbc60211340" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 38987 } } peers { permanent_uuid: "0431291fa0624d708fe8d5f41049cc41" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35561 } }
I20260928 16:42:05.763456 6746 raft_consensus.cc:374] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 0431291fa0624d708fe8d5f41049cc41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:05.763501 6378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9a8ba1cbb074d4c9ae645a80e0adc56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:05.763562 6746 raft_consensus.cc:729] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 0431291fa0624d708fe8d5f41049cc41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0431291fa0624d708fe8d5f41049cc41, State: Initialized, Role: FOLLOWER
I20260928 16:42:05.764169 6746 consensus_queue.cc:260] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 0431291fa0624d708fe8d5f41049cc41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fa3182455a4c398a3563d5fc49c237" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 38683 } } peers { permanent_uuid: "173e35deb6ac4e69bacc5bbc60211340" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 38987 } } peers { permanent_uuid: "0431291fa0624d708fe8d5f41049cc41" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35561 } }
I20260928 16:42:05.765559 6746 ts_tablet_manager.cc:1428] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 0431291fa0624d708fe8d5f41049cc41: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260928 16:42:05.767021 6720 heartbeater.cc:503] Master 127.25.71.126:35713 was elected leader, sending a full tablet report...
I20260928 16:42:05.784758 6749 tablet_bootstrap.cc:492] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340: Bootstrap starting.
I20260928 16:42:05.789460 6749 tablet_bootstrap.cc:654] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340: Neither blocks nor log segments found. Creating new log.
I20260928 16:42:05.791023 6749 log.cc:824] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:05.796731 6749 tablet_bootstrap.cc:492] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340: No bootstrap required, opened a new log
I20260928 16:42:05.797116 6749 ts_tablet_manager.cc:1397] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20260928 16:42:05.797963 6750 tablet_bootstrap.cc:492] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 99fa3182455a4c398a3563d5fc49c237: Bootstrap starting.
I20260928 16:42:05.801196 6750 tablet_bootstrap.cc:654] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 99fa3182455a4c398a3563d5fc49c237: Neither blocks nor log segments found. Creating new log.
I20260928 16:42:05.802361 6750 log.cc:824] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 99fa3182455a4c398a3563d5fc49c237: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:05.803531 6749 raft_consensus.cc:348] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fa3182455a4c398a3563d5fc49c237" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 38683 } } peers { permanent_uuid: "173e35deb6ac4e69bacc5bbc60211340" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 38987 } } peers { permanent_uuid: "0431291fa0624d708fe8d5f41049cc41" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35561 } }
I20260928 16:42:05.804142 6749 raft_consensus.cc:374] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:05.804265 6749 raft_consensus.cc:729] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 173e35deb6ac4e69bacc5bbc60211340, State: Initialized, Role: FOLLOWER
I20260928 16:42:05.805068 6749 consensus_queue.cc:260] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fa3182455a4c398a3563d5fc49c237" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 38683 } } peers { permanent_uuid: "173e35deb6ac4e69bacc5bbc60211340" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 38987 } } peers { permanent_uuid: "0431291fa0624d708fe8d5f41049cc41" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35561 } }
I20260928 16:42:05.805802 6750 tablet_bootstrap.cc:492] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 99fa3182455a4c398a3563d5fc49c237: No bootstrap required, opened a new log
I20260928 16:42:05.806085 6750 ts_tablet_manager.cc:1397] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 99fa3182455a4c398a3563d5fc49c237: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.007s
I20260928 16:42:05.806793 6749 ts_tablet_manager.cc:1428] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340: Time spent starting tablet: real 0.010s user 0.008s sys 0.001s
I20260928 16:42:05.807314 6580 heartbeater.cc:503] Master 127.25.71.126:35713 was elected leader, sending a full tablet report...
I20260928 16:42:05.810303 6750 raft_consensus.cc:348] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 99fa3182455a4c398a3563d5fc49c237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fa3182455a4c398a3563d5fc49c237" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 38683 } } peers { permanent_uuid: "173e35deb6ac4e69bacc5bbc60211340" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 38987 } } peers { permanent_uuid: "0431291fa0624d708fe8d5f41049cc41" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35561 } }
I20260928 16:42:05.810752 6750 raft_consensus.cc:374] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 99fa3182455a4c398a3563d5fc49c237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:05.810822 6750 raft_consensus.cc:729] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 99fa3182455a4c398a3563d5fc49c237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99fa3182455a4c398a3563d5fc49c237, State: Initialized, Role: FOLLOWER
I20260928 16:42:05.811415 6750 consensus_queue.cc:260] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 99fa3182455a4c398a3563d5fc49c237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fa3182455a4c398a3563d5fc49c237" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 38683 } } peers { permanent_uuid: "173e35deb6ac4e69bacc5bbc60211340" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 38987 } } peers { permanent_uuid: "0431291fa0624d708fe8d5f41049cc41" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35561 } }
I20260928 16:42:05.812691 6750 ts_tablet_manager.cc:1428] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 99fa3182455a4c398a3563d5fc49c237: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:42:05.813172 6445 heartbeater.cc:503] Master 127.25.71.126:35713 was elected leader, sending a full tablet report...
I20260928 16:42:05.854689 6753 raft_consensus.cc:482] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:42:05.855041 6753 raft_consensus.cc:504] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fa3182455a4c398a3563d5fc49c237" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 38683 } } peers { permanent_uuid: "173e35deb6ac4e69bacc5bbc60211340" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 38987 } } peers { permanent_uuid: "0431291fa0624d708fe8d5f41049cc41" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35561 } }
I20260928 16:42:05.856943 6753 leader_election.cc:288] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99fa3182455a4c398a3563d5fc49c237 (127.25.71.65:38683), 0431291fa0624d708fe8d5f41049cc41 (127.25.71.67:35561)
Sep 28 16:42:05 dist-test-slave-nqgh krb5kdc[5250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613725, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for HTTP/127.25.71.84@KRBTEST.COM
I20260928 16:42:05.864377 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:05.858151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:37089 (local address 127.25.71.67:35561)
0928 16:42:05.858317 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:05.858323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:05.858703 (+ 380us) server_negotiation.cc:407] Connection header received
0928 16:42:05.859606 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:05.859610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:05.859667 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:05.859718 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:05.860142 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.860948 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.862967 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.863603 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.863713 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:05.863770 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:05.864119 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 16:42:05.864212 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:42:05.865291 6755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:05.857815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:35561 (local address 127.25.71.66:37089)
0928 16:42:05.858503 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:05.858519 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:05.859435 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:05.859862 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:05.859865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:05.859883 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:05.860013 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:05.860018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.861130 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.861133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:05.862783 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:05.862797 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.863872 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.863876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:05.863980 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:05.864006 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:42:05.864962 (+ 956us) client_negotiation.cc:240] Negotiation successful
0928 16:42:05.865079 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":70,"thread_start_us":113,"threads_started":1}
I20260928 16:42:05.866348 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a8ba1cbb074d4c9ae645a80e0adc56" candidate_uuid: "173e35deb6ac4e69bacc5bbc60211340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0431291fa0624d708fe8d5f41049cc41" is_pre_election: true
I20260928 16:42:05.866811 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:05.858247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:51641 (local address 127.25.71.65:38683)
0928 16:42:05.858430 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:05.858435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:05.858454 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:42:05.858745 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:05.858748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:05.858804 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:05.858856 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:05.859434 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.860307 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.865173 (+ 4866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.865850 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.865978 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:05.866051 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:05.866470 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 16:42:05.866590 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:42:05.866858 6659 raft_consensus.cc:2454] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 0431291fa0624d708fe8d5f41049cc41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 173e35deb6ac4e69bacc5bbc60211340 in term 0.
I20260928 16:42:05.867399 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:05.857301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:38683 (local address 127.25.71.66:51641)
0928 16:42:05.858199 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:05.858216 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:05.858559 (+ 343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:05.859043 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:05.859046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:05.859068 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:05.859261 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:05.859268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.860497 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.860501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:05.864934 (+ 4433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:05.864955 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.866188 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.866192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:05.866307 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:05.866336 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 16:42:05.867108 (+ 772us) client_negotiation.cc:240] Negotiation successful
0928 16:42:05.867227 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":155,"threads_started":1}
I20260928 16:42:05.867640 6467 leader_election.cc:302] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0431291fa0624d708fe8d5f41049cc41, 173e35deb6ac4e69bacc5bbc60211340; no voters:
I20260928 16:42:05.868136 6753 raft_consensus.cc:2781] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:42:05.868224 6753 raft_consensus.cc:482] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:42:05.868260 6753 raft_consensus.cc:3037] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:05.869616 6753 raft_consensus.cc:504] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fa3182455a4c398a3563d5fc49c237" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 38683 } } peers { permanent_uuid: "173e35deb6ac4e69bacc5bbc60211340" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 38987 } } peers { permanent_uuid: "0431291fa0624d708fe8d5f41049cc41" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35561 } }
I20260928 16:42:05.869669 6388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a8ba1cbb074d4c9ae645a80e0adc56" candidate_uuid: "173e35deb6ac4e69bacc5bbc60211340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99fa3182455a4c398a3563d5fc49c237" is_pre_election: true
I20260928 16:42:05.870085 6753 leader_election.cc:288] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 [CANDIDATE]: Term 1 election: Requested vote from peers 99fa3182455a4c398a3563d5fc49c237 (127.25.71.65:38683), 0431291fa0624d708fe8d5f41049cc41 (127.25.71.67:35561)
I20260928 16:42:05.870059 6388 raft_consensus.cc:2454] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 99fa3182455a4c398a3563d5fc49c237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 173e35deb6ac4e69bacc5bbc60211340 in term 0.
I20260928 16:42:05.870714 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a8ba1cbb074d4c9ae645a80e0adc56" candidate_uuid: "173e35deb6ac4e69bacc5bbc60211340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0431291fa0624d708fe8d5f41049cc41"
I20260928 16:42:05.870745 6388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a8ba1cbb074d4c9ae645a80e0adc56" candidate_uuid: "173e35deb6ac4e69bacc5bbc60211340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99fa3182455a4c398a3563d5fc49c237"
I20260928 16:42:05.870880 6388 raft_consensus.cc:3037] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 99fa3182455a4c398a3563d5fc49c237 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:05.870877 6659 raft_consensus.cc:3037] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 0431291fa0624d708fe8d5f41049cc41 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:05.872296 6659 raft_consensus.cc:2454] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 0431291fa0624d708fe8d5f41049cc41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 173e35deb6ac4e69bacc5bbc60211340 in term 1.
I20260928 16:42:05.872556 6388 raft_consensus.cc:2454] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 99fa3182455a4c398a3563d5fc49c237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 173e35deb6ac4e69bacc5bbc60211340 in term 1.
I20260928 16:42:05.873438 6467 leader_election.cc:302] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 173e35deb6ac4e69bacc5bbc60211340, 99fa3182455a4c398a3563d5fc49c237; no voters:
I20260928 16:42:05.873965 6753 raft_consensus.cc:2781] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:42:05.874258 6753 raft_consensus.cc:686] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 [term 1 LEADER]: Becoming Leader. State: Replica: 173e35deb6ac4e69bacc5bbc60211340, State: Running, Role: LEADER
I20260928 16:42:05.874773 6753 consensus_queue.cc:237] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fa3182455a4c398a3563d5fc49c237" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 38683 } } peers { permanent_uuid: "173e35deb6ac4e69bacc5bbc60211340" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 38987 } } peers { permanent_uuid: "0431291fa0624d708fe8d5f41049cc41" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35561 } }
W20260928 16:42:05.876111 6581 tablet.cc:2369] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:42:05.878594 6247 catalog_manager.cc:5495] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 reported cstate change: term changed from 0 to 1, leader changed from <none> to 173e35deb6ac4e69bacc5bbc60211340 (127.25.71.66). New cstate: current_term: 1 leader_uuid: "173e35deb6ac4e69bacc5bbc60211340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fa3182455a4c398a3563d5fc49c237" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 38683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "173e35deb6ac4e69bacc5bbc60211340" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0431291fa0624d708fe8d5f41049cc41" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35561 } health_report { overall_health: UNKNOWN } } }
I20260928 16:42:05.917243 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:05.911340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47922 (local address 127.25.71.66:38987)
0928 16:42:05.911487 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:05.911492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:05.911796 (+ 304us) server_negotiation.cc:407] Connection header received
0928 16:42:05.911931 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:05.911934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:05.912011 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:05.912122 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:42:05.912434 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.913215 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.914520 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:05.914920 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:05.915045 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:05.915597 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:42:05.916413 (+ 816us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:42:05.916574 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:05.916701 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 16:42:05.916799 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:42:05.925256 6659 raft_consensus.cc:1261] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 0431291fa0624d708fe8d5f41049cc41 [term 1 FOLLOWER]: Refusing update from remote peer 173e35deb6ac4e69bacc5bbc60211340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:42:05.925568 6388 raft_consensus.cc:1261] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 99fa3182455a4c398a3563d5fc49c237 [term 1 FOLLOWER]: Refusing update from remote peer 173e35deb6ac4e69bacc5bbc60211340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:42:05.926349 6753 consensus_queue.cc:1035] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0431291fa0624d708fe8d5f41049cc41" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:42:05.926697 6757 consensus_queue.cc:1035] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99fa3182455a4c398a3563d5fc49c237" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 38683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:42:05.943100 6760 mvcc.cc:205] Tried to move back new op lower bound from 7334353821379907584 to 7334353821186207744. Current Snapshot: MvccSnapshot[applied={T|T < 7334353821379907584}]
I20260928 16:42:05.961350 6762 mvcc.cc:205] Tried to move back new op lower bound from 7334353821379907584 to 7334353821186207744. Current Snapshot: MvccSnapshot[applied={T|T < 7334353821379907584}]
I20260928 16:42:05.962206 6247 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:42:05.962436 6247 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:42:05.966184 6247 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40d30c200c4b4935bfa2bdedd1edf9a2: Sending DeleteTablet for 3 replicas of tablet b9a8ba1cbb074d4c9ae645a80e0adc56
I20260928 16:42:05.968026 6378 tablet_service.cc:1514] Processing DeleteTablet for tablet b9a8ba1cbb074d4c9ae645a80e0adc56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:42:05 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:34908
I20260928 16:42:05.968704 6654 tablet_service.cc:1514] Processing DeleteTablet for tablet b9a8ba1cbb074d4c9ae645a80e0adc56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:42:05 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:54340
I20260928 16:42:05.969918 6514 tablet_service.cc:1514] Processing DeleteTablet for tablet b9a8ba1cbb074d4c9ae645a80e0adc56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:42:05 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:47916
I20260928 16:42:05.971390 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 6285
I20260928 16:42:05.972126 6761 mvcc.cc:205] Tried to move back new op lower bound from 7334353821379907584 to 7334353821186207744. Current Snapshot: MvccSnapshot[applied={T|T < 7334353821379907584 or (T in {7334353821379907584})}]
I20260928 16:42:05.973518 6771 tablet_replica.cc:330] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340: stopping tablet replica
I20260928 16:42:05.973886 6770 tablet_replica.cc:330] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 0431291fa0624d708fe8d5f41049cc41: stopping tablet replica
I20260928 16:42:05.974427 6770 raft_consensus.cc:2229] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 0431291fa0624d708fe8d5f41049cc41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:42:05.974915 6771 raft_consensus.cc:2229] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:42:05.975587 6771 raft_consensus.cc:2258] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 173e35deb6ac4e69bacc5bbc60211340 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:42:05.976132 6770 raft_consensus.cc:2258] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 0431291fa0624d708fe8d5f41049cc41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:42:05.978473 6770 ts_tablet_manager.cc:1905] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 0431291fa0624d708fe8d5f41049cc41: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 16:42:05.982939 6231 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.71.65:38683 (error 108)
I20260928 16:42:05.984247 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 6449
I20260928 16:42:05.988476 6770 ts_tablet_manager.cc:1918] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 0431291fa0624d708fe8d5f41049cc41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:42:05.988732 6770 log.cc:1196] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 0431291fa0624d708fe8d5f41049cc41: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/b9a8ba1cbb074d4c9ae645a80e0adc56
I20260928 16:42:05.989099 6770 ts_tablet_manager.cc:1939] T b9a8ba1cbb074d4c9ae645a80e0adc56 P 0431291fa0624d708fe8d5f41049cc41: Deleting consensus metadata
I20260928 16:42:05.991712 6231 catalog_manager.cc:4841] TS 0431291fa0624d708fe8d5f41049cc41 (127.25.71.67:35561): tablet b9a8ba1cbb074d4c9ae645a80e0adc56 (table test-table [id=931f3b46300445abb81161de59ef9db8]) successfully deleted
I20260928 16:42:05.991936 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:05.991437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:38683 (local address 127.0.0.1:34912)
0928 16:42:05.991671 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:05.991816 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.65:38683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260928 16:42:05.992385 6231 catalog_manager.cc:4584] TS 99fa3182455a4c398a3563d5fc49c237 (127.25.71.65:38683): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b9a8ba1cbb074d4c9ae645a80e0adc56: Network error: Client connection negotiation failed: client connection to 127.25.71.65:38683: connect: Connection refused (error 111)
I20260928 16:42:05.995112 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:05.994758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:38987 (local address 127.0.0.1:47928)
0928 16:42:05.994929 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:05.995010 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.66:38987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 16:42:05.995229 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 6583
I20260928 16:42:06.002493 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 6214
I20260928 16:42:06.011010 25885 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 16:42:06.358280 25885 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 16:42:06.358417 25885 mini_ranger.cc:66] Stopping Ranger...
I20260928 16:42:06.704748 25885 mini_ranger.cc:68] Stopped Ranger
2026-09-28 16:42:06.705 UTC [5280] LOG: received smart shutdown request
2026-09-28 16:42:06.710 UTC [5280] LOG: background worker "logical replication launcher" (PID 5289) exited with exit code 1
2026-09-28 16:42:06.710 UTC [5284] LOG: shutting down
2026-09-28 16:42:06.962 UTC [5280] LOG: database system is shut down
2026-09-28T16:42:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (58966 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:42:07 dist-test-slave-nqgh krb5kdc[6781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:42:07 dist-test-slave-nqgh krb5kdc[6781](info): set up 2 sockets
Sep 28 16:42:07 dist-test-slave-nqgh krb5kdc[6781](info): commencing operation
krb5kdc: starting...
W20260928 16:42:08.080531 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:42:08 dist-test-slave-nqgh krb5kdc[6781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:42:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:42:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:42:08.223742 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:39899
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:34209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:39899,127.25.71.125:38871,127.25.71.124:43681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:42:08.364360 6794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:08.364773 6794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:08.364842 6794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:08.369994 6794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:42:08.370150 6794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:08.370184 6794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:08.370208 6794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:42:08.370229 6794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:42:08.377368 6794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:34209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:39899,127.25.71.125:38871,127.25.71.124:43681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:39899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:08.379024 6794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:08.380501 6794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:08.391855 6799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:08.391887 6802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:08.391855 6800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:08.392539 6794 server_base.cc:1034] running on GCE node
I20260928 16:42:08.393292 6794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:08.394845 6794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:08.396065 6794 hybrid_clock.cc:648] HybridClock initialized: now 1790613728396022 us; error 65 us; skew 500 ppm
Sep 28 16:42:08 dist-test-slave-nqgh krb5kdc[6781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613728, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:08.399904 6794 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:42:08.401257 6794 webserver.cc:460] Webserver started at http://127.25.71.126:44993/ using document root <none> and password file <none>
I20260928 16:42:08.401983 6794 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:08.402070 6794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:08.402294 6794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:42:08.405210 6794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "6e8c48c5784b484f9f576d95745f747e"
format_stamp: "Formatted at 2026-09-28 16:42:08 on dist-test-slave-nqgh"
server_key: "1f3f324e5e7d14f1479534bc99b89155"
server_key_iv: "bd3725e0b2e639b3b00f8eaacd85831c"
server_key_version: "encryptionkey@0"
I20260928 16:42:08.405937 6794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "6e8c48c5784b484f9f576d95745f747e"
format_stamp: "Formatted at 2026-09-28 16:42:08 on dist-test-slave-nqgh"
server_key: "1f3f324e5e7d14f1479534bc99b89155"
server_key_iv: "bd3725e0b2e639b3b00f8eaacd85831c"
server_key_version: "encryptionkey@0"
I20260928 16:42:08.411257 6794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 16:42:08.415138 6809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:08.417160 6794 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:42:08.417361 6794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "6e8c48c5784b484f9f576d95745f747e"
format_stamp: "Formatted at 2026-09-28 16:42:08 on dist-test-slave-nqgh"
server_key: "1f3f324e5e7d14f1479534bc99b89155"
server_key_iv: "bd3725e0b2e639b3b00f8eaacd85831c"
server_key_version: "encryptionkey@0"
I20260928 16:42:08.417519 6794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:08.445101 6794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:08.449961 6794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:08.450205 6794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:08.459715 6794 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:39899
I20260928 16:42:08.459743 6861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:39899 every 8 connection(s)
I20260928 16:42:08.461061 6794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:42:08.463419 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 6794
I20260928 16:42:08.463577 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:42:08.463908 25885 external_mini_cluster.cc:1428] Setting key 3515186474573edb6dbf1e96b392bb7f
I20260928 16:42:08.465205 6862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:08.469678 6862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.125@KRBTEST.COM" created.
Sep 28 16:42:08 dist-test-slave-nqgh krb5kdc[6781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613728, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:08.489815 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.472993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:39899 (local address 127.0.0.1:43544)
0928 16:42:08.473858 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.473919 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:08.474974 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:08.476054 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:08.476060 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:08.476593 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:08.476947 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:08.476965 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.478267 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.478269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:08.479118 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:08.479130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.479806 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.479810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:08.479955 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:08.480551 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:08.480570 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:08.482932 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:08.485483 (+ 2551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:08.485503 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:08.485523 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:08.485972 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:08.486431 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:08.486434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:08.486435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:08.486585 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:08.487503 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:08.487513 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:08.487749 (+ 236us) client_negotiation.cc:769] Sending connection context
0928 16:42:08.487952 (+ 203us) client_negotiation.cc:240] Negotiation successful
0928 16:42:08.488609 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":200,"thread_start_us":210,"threads_started":1}
I20260928 16:42:08.489817 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.474328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43544 (local address 127.25.71.126:39899)
0928 16:42:08.474722 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:08.474733 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:08.474782 (+ 49us) server_negotiation.cc:407] Connection header received
0928 16:42:08.475375 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:08.475381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:08.475745 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:08.475971 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:08.477127 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.478132 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.479235 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.479641 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.479799 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:08.483197 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:08.483205 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:08.483220 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:08.483250 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:08.485288 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:08.486173 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:08.486180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:08.486181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:08.486329 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:08.486827 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:08.486831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:08.486832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:08.487335 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:08.487470 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:08.488100 (+ 630us) server_negotiation.cc:299] Negotiation successful
0928 16:42:08.488492 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":245,"thread_start_us":112,"threads_started":1}
I20260928 16:42:08.494132 6862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:42:08.495280 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.494818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38871 (local address 127.0.0.1:55960)
0928 16:42:08.495022 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.495153 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 16:42:08.495646 6811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38871: connect: Connection refused (error 111)
I20260928 16:42:08.496905 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.496602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38871 (local address 127.0.0.1:55972)
0928 16:42:08.496740 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.496803 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 16:42:08.497059 6862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.125:38871: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38871: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.125.keytab.
Entry for principal kudu/127.25.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.125.keytab.
I20260928 16:42:08.499114 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.125:38871
--webserver_interface=127.25.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.125.keytab
--principal=kudu/127.25.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:34209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:39899,127.25.71.125:38871,127.25.71.124:43681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:42:08.546420 6862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } attempt: 1
I20260928 16:42:08.547808 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.547309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38871 (local address 127.0.0.1:55980)
0928 16:42:08.547558 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.547675 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260928 16:42:08.548786 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.548498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38871 (local address 127.0.0.1:55992)
0928 16:42:08.548632 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.548704 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 16:42:08.548977 6862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.125:38871: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38871: connect: Connection refused (error 111)
I20260928 16:42:08.617229 6862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } attempt: 2
I20260928 16:42:08.618906 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.618328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38871 (local address 127.0.0.1:56006)
0928 16:42:08.618593 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.618758 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260928 16:42:08.620069 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.619719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38871 (local address 127.0.0.1:56022)
0928 16:42:08.619876 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.619965 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 16:42:08.620321 6862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.125:38871: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38871: connect: Connection refused (error 111)
W20260928 16:42:08.643361 6869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:08.643800 6869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:08.643867 6869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:08.648792 6869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:42:08.648926 6869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:08.648957 6869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:08.648981 6869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:42:08.649001 6869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:42:08.656112 6869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:34209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:39899,127.25.71.125:38871,127.25.71.124:43681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.125.keytab
--principal=kudu/127.25.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.125:38871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.71.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:08.657840 6869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:08.659358 6869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:08.670423 6874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:08.670421 6877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:08.670547 6875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:08.670966 6869 server_base.cc:1034] running on GCE node
I20260928 16:42:08.671789 6869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:08.673456 6869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:08.674692 6869 hybrid_clock.cc:648] HybridClock initialized: now 1790613728674667 us; error 47 us; skew 500 ppm
Sep 28 16:42:08 dist-test-slave-nqgh krb5kdc[6781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613728, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:08.678292 6869 init.cc:377] Logged in from keytab as kudu/127.25.71.125@KRBTEST.COM (short username kudu)
I20260928 16:42:08.679576 6869 webserver.cc:460] Webserver started at http://127.25.71.125:36715/ using document root <none> and password file <none>
I20260928 16:42:08.680315 6869 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:08.680387 6869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:08.680627 6869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:42:08.683313 6869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-1/data/instance:
uuid: "daa5636d6076402d88ee9ccf2e742a72"
format_stamp: "Formatted at 2026-09-28 16:42:08 on dist-test-slave-nqgh"
server_key: "75b887f339da309cb27a973e35253079"
server_key_iv: "6cf3322c64ceedde5b7183666f676577"
server_key_version: "encryptionkey@0"
I20260928 16:42:08.684000 6869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-1/wal/instance:
uuid: "daa5636d6076402d88ee9ccf2e742a72"
format_stamp: "Formatted at 2026-09-28 16:42:08 on dist-test-slave-nqgh"
server_key: "75b887f339da309cb27a973e35253079"
server_key_iv: "6cf3322c64ceedde5b7183666f676577"
server_key_version: "encryptionkey@0"
I20260928 16:42:08.689208 6869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:42:08.693107 6884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:08.695174 6869 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:42:08.695389 6869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-1/wal
uuid: "daa5636d6076402d88ee9ccf2e742a72"
format_stamp: "Formatted at 2026-09-28 16:42:08 on dist-test-slave-nqgh"
server_key: "75b887f339da309cb27a973e35253079"
server_key_iv: "6cf3322c64ceedde5b7183666f676577"
server_key_version: "encryptionkey@0"
I20260928 16:42:08.695562 6869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:08.711584 6862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } attempt: 3
I20260928 16:42:08.712765 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.712395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38871 (local address 127.0.0.1:56028)
0928 16:42:08.712559 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.712652 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 16:42:08.713965 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.713644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38871 (local address 127.0.0.1:56034)
0928 16:42:08.713817 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.713882 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 16:42:08.714246 6862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.125:38871: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38871: connect: Connection refused (error 111)
I20260928 16:42:08.715655 6869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:08.719154 6869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:08.719363 6869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:08.727973 6869 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.125:38871
I20260928 16:42:08.728084 6936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.125:38871 every 8 connection(s)
I20260928 16:42:08.729148 6869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-1/data/info.pb
I20260928 16:42:08.733134 6937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:08.738370 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 6869
I20260928 16:42:08.738498 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-1/wal/instance
I20260928 16:42:08.738829 25885 external_mini_cluster.cc:1428] Setting key 5f92add913f01ab69850bd141f0f1a53
I20260928 16:42:08.738621 6937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.71.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.124@KRBTEST.COM" created.
Sep 28 16:42:08 dist-test-slave-nqgh krb5kdc[6781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613728, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.125@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:08.757558 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.741609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43552 (local address 127.25.71.126:39899)
0928 16:42:08.741853 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:08.741859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:08.742928 (+ 1069us) server_negotiation.cc:407] Connection header received
0928 16:42:08.744119 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:08.744124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:08.744217 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:08.744372 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:08.745745 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.746662 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.748017 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.748303 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.748416 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:08.752841 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:08.752846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:08.752849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:08.752901 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:08.754591 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:08.755409 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:08.755414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:08.755416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:08.755543 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:08.756012 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:08.756016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:08.756018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:08.756329 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:08.756450 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:08.757152 (+ 702us) server_negotiation.cc:299] Negotiation successful
0928 16:42:08.757339 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 16:42:08.758390 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.742090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:39899 (local address 127.0.0.1:43552)
0928 16:42:08.742710 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.742783 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:08.743858 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:08.744597 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:08.744606 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:08.745005 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:08.745559 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:08.745573 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.746837 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.746840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:08.747867 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:08.747880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.748486 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.748489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:08.748665 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:08.749285 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:08.749304 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:08.752489 (+ 3185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:08.754868 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:08.754877 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:08.754896 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:08.755242 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:08.755685 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:08.755688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:08.755690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:08.755892 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:08.756506 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:08.756515 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:08.756840 (+ 325us) client_negotiation.cc:769] Sending connection context
0928 16:42:08.757041 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 16:42:08.757362 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":136,"threads_started":1}
I20260928 16:42:08.760414 6937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:42:08 dist-test-slave-nqgh krb5kdc[6781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613728, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.125@KRBTEST.COM for kudu/127.25.71.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.124.keytab.
Entry for principal kudu/127.25.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.124.keytab.
I20260928 16:42:08.771684 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.124:43681
--webserver_interface=127.25.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.124.keytab
--principal=kudu/127.25.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:34209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:39899,127.25.71.125:38871,127.25.71.124:43681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:42:08.772863 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.761174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38871 (local address 127.0.0.1:56044)
0928 16:42:08.761300 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.761315 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:08.761404 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:08.762591 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:08.762597 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:08.762941 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:08.763087 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:08.763095 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.764506 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.764510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:08.765140 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:08.765152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.766079 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.766082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:08.766196 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:08.766877 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:08.766893 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:08.768346 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:08.770806 (+ 2460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:08.770810 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:08.770812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:08.771126 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:08.771641 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:08.771644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:08.771645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:08.771730 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:08.772297 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:08.772300 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:08.772389 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 16:42:08.772478 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 16:42:08.772648 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 16:42:08.772886 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.761359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56044 (local address 127.25.71.125:38871)
0928 16:42:08.761941 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:08.761955 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:08.761985 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:42:08.762112 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:08.762118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:08.762369 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:08.762522 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:08.763263 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.764321 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.765276 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.765915 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.766021 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:08.768596 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:08.768609 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:08.768629 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:08.768664 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:08.770662 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:08.771318 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:08.771329 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:08.771331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:08.771505 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:08.771826 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:08.771830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:08.771832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:08.772191 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:08.772337 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:08.772595 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 16:42:08.772745 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":220,"threads_started":1}
I20260928 16:42:08.775672 6937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:42:08.776753 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.776386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:43681 (local address 127.0.0.1:34946)
0928 16:42:08.776571 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.776658 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:43681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 16:42:08.777117 6885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.71.124:43681: connect: Connection refused (error 111)
I20260928 16:42:08.778357 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.778103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:43681 (local address 127.0.0.1:34952)
0928 16:42:08.778232 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.778286 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:43681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 16:42:08.778606 6937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.124:43681: Network error: Client connection negotiation failed: client connection to 127.25.71.124:43681: connect: Connection refused (error 111)
I20260928 16:42:08.828008 6937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } attempt: 1
I20260928 16:42:08.829176 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.828828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:43681 (local address 127.0.0.1:34960)
0928 16:42:08.829003 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.829081 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:43681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 16:42:08.830302 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.829988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:43681 (local address 127.0.0.1:34970)
0928 16:42:08.830178 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.830234 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:43681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 16:42:08.830466 6937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.124:43681: Network error: Client connection negotiation failed: client connection to 127.25.71.124:43681: connect: Connection refused (error 111)
I20260928 16:42:08.857476 6862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } attempt: 4
Sep 28 16:42:08 dist-test-slave-nqgh krb5kdc[6781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613728, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for kudu/127.25.71.125@KRBTEST.COM
I20260928 16:42:08.870329 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.858309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38871 (local address 127.0.0.1:56046)
0928 16:42:08.858540 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.858560 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:08.858711 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:08.859071 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:08.859073 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:08.859359 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:08.859522 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:08.859528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.860695 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.860699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:08.861814 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:08.861825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.862531 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.862535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:08.862758 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:08.863645 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:08.863663 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:08.865259 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:08.868552 (+ 3293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:08.868556 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:08.868558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:08.868855 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:08.869209 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:08.869211 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:08.869212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:08.869293 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:08.869757 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:08.869760 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:08.869896 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 16:42:08.869995 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 16:42:08.870158 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260928 16:42:08.870424 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.858460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.25.71.125:38871)
0928 16:42:08.858679 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:08.858684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:08.858709 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:42:08.858836 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:08.858840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:08.858906 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:08.859030 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:08.859666 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.860498 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.861963 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.862354 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.862478 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:08.865459 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:08.865464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:08.865467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:08.865509 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:08.868330 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:08.868987 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:08.868992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:08.868994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:08.869104 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:08.869386 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:08.869389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:08.869391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:08.869646 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:08.869758 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:08.870081 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 16:42:08.870226 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 16:42:08.871553 6862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:42:08.872643 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.872293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:43681 (local address 127.0.0.1:34980)
0928 16:42:08.872464 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.872536 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:43681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 16:42:08.873591 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.873290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:43681 (local address 127.0.0.1:34994)
0928 16:42:08.873449 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.873514 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:43681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260928 16:42:08.873785 6862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.124:43681: Network error: Client connection negotiation failed: client connection to 127.25.71.124:43681: connect: Connection refused (error 111)
I20260928 16:42:08.898687 6937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } attempt: 2
I20260928 16:42:08.899744 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.899395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:43681 (local address 127.0.0.1:35008)
0928 16:42:08.899567 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.899642 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:43681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 16:42:08.900790 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.900536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:43681 (local address 127.0.0.1:35024)
0928 16:42:08.900660 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.900722 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:43681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 16:42:08.900926 6937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.124:43681: Network error: Client connection negotiation failed: client connection to 127.25.71.124:43681: connect: Connection refused (error 111)
W20260928 16:42:08.901309 6943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:08.901618 6943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:08.901698 6943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:08.906116 6943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:42:08.906205 6943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:08.906225 6943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:08.906242 6943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:42:08.906257 6943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:42:08.912159 6943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:34209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:39899,127.25.71.125:38871,127.25.71.124:43681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.124.keytab
--principal=kudu/127.25.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.124:43681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.71.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:08.913427 6943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:08.914567 6943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:08.924659 6950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:08.924670 6952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:08.924849 6943 server_base.cc:1034] running on GCE node
W20260928 16:42:08.924677 6949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:08.925743 6943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:08.927126 6943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:08.928380 6943 hybrid_clock.cc:648] HybridClock initialized: now 1790613728928344 us; error 63 us; skew 500 ppm
Sep 28 16:42:08 dist-test-slave-nqgh krb5kdc[6781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613728, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:08.932073 6943 init.cc:377] Logged in from keytab as kudu/127.25.71.124@KRBTEST.COM (short username kudu)
I20260928 16:42:08.932969 6862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } attempt: 1
I20260928 16:42:08.933269 6943 webserver.cc:460] Webserver started at http://127.25.71.124:39999/ using document root <none> and password file <none>
I20260928 16:42:08.933947 6943 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:08.933974 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.933633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:43681 (local address 127.0.0.1:35028)
0928 16:42:08.933804 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.933873 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:43681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 16:42:08.934034 6943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:08.934314 6943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:42:08.934777 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.934520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:43681 (local address 127.0.0.1:35044)
0928 16:42:08.934658 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.934715 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:43681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 16:42:08.934898 6862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.124:43681: Network error: Client connection negotiation failed: client connection to 127.25.71.124:43681: connect: Connection refused (error 111)
I20260928 16:42:08.936709 6943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-2/data/instance:
uuid: "2a7b4201fe4545b5971ce7ab062fb5b8"
format_stamp: "Formatted at 2026-09-28 16:42:08 on dist-test-slave-nqgh"
server_key: "44ebdd9ff8d1863426916d741fe5b855"
server_key_iv: "a31fb6a5ed10725bf001eb73d924f133"
server_key_version: "encryptionkey@0"
I20260928 16:42:08.937325 6943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-2/wal/instance:
uuid: "2a7b4201fe4545b5971ce7ab062fb5b8"
format_stamp: "Formatted at 2026-09-28 16:42:08 on dist-test-slave-nqgh"
server_key: "44ebdd9ff8d1863426916d741fe5b855"
server_key_iv: "a31fb6a5ed10725bf001eb73d924f133"
server_key_version: "encryptionkey@0"
I20260928 16:42:08.942041 6943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 16:42:08.945495 6959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:08.947429 6943 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:42:08.947625 6943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-2/wal
uuid: "2a7b4201fe4545b5971ce7ab062fb5b8"
format_stamp: "Formatted at 2026-09-28 16:42:08 on dist-test-slave-nqgh"
server_key: "44ebdd9ff8d1863426916d741fe5b855"
server_key_iv: "a31fb6a5ed10725bf001eb73d924f133"
server_key_version: "encryptionkey@0"
I20260928 16:42:08.947770 6943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:08.960922 6943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:08.964691 6943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:08.964907 6943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:08.973203 6943 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.124:43681
I20260928 16:42:08.973253 7011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.124:43681 every 8 connection(s)
I20260928 16:42:08.974301 6943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-2/data/info.pb
I20260928 16:42:08.978416 7012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:08.979897 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 6943
I20260928 16:42:08.980058 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613545667819-25885-0/minicluster-data/master-2/wal/instance
I20260928 16:42:08.980432 25885 external_mini_cluster.cc:1428] Setting key 6ec1f7b5d2fbac1e0cbb475e35cf927f
I20260928 16:42:08.984346 7012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:42:08 dist-test-slave-nqgh krb5kdc[6781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.124@KRBTEST.COM
I20260928 16:42:08.992089 6937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } attempt: 3
Sep 28 16:42:08 dist-test-slave-nqgh krb5kdc[6781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613728, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.124@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:08.998175 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.987019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43568 (local address 127.25.71.126:39899)
0928 16:42:08.987230 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:08.987235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:08.987732 (+ 497us) server_negotiation.cc:407] Connection header received
0928 16:42:08.987924 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:08.987926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:08.987981 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:08.988061 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:08.989218 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.989755 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.990746 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.990969 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.991061 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:08.994074 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:08.994078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:08.994080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:08.994108 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:08.995750 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:08.996411 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:08.996416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:08.996418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:08.996530 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:08.996816 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:08.996820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:08.996822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:08.997097 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:08.997168 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:08.997847 (+ 679us) server_negotiation.cc:299] Negotiation successful
0928 16:42:08.998011 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 16:42:08.998571 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.982224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.25.71.124:43681)
0928 16:42:08.982758 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:08.982777 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:08.982819 (+ 42us) server_negotiation.cc:407] Connection header received
0928 16:42:08.983661 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:08.983683 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:08.984097 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:08.984436 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:08.985583 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.986873 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.988271 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.988941 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.989095 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:08.991997 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:08.992005 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:08.992021 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:08.992044 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:08.994302 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:08.994952 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:08.994960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:08.994970 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:08.995166 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:08.995527 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:08.995530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:08.995532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:08.996201 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:08.996422 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:08.996823 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 16:42:08.997190 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":136,"threads_started":1}
Sep 28 16:42:08 dist-test-slave-nqgh krb5kdc[6781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613728, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.125@KRBTEST.COM for kudu/127.25.71.124@KRBTEST.COM
I20260928 16:42:08.998883 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.986940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:39899 (local address 127.0.0.1:43568)
0928 16:42:08.987529 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.987613 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:08.987796 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:08.988243 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:08.988250 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:08.988808 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:08.989048 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:08.989060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.989899 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.989902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:08.990584 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:08.990594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.991125 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.991136 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:08.991242 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:08.991825 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:08.991851 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:08.993892 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:08.995897 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:08.995910 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:08.995919 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:08.996270 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:08.996636 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:08.996639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:08.996641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:08.996727 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:08.997416 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:08.997424 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:08.997559 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 16:42:08.997728 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 16:42:08.997969 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":170,"threads_started":1}
I20260928 16:42:09.000788 7012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:42:09.002184 6862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } attempt: 2
I20260928 16:42:09.003716 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.992744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:43681 (local address 127.0.0.1:35072)
0928 16:42:08.993009 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:08.993026 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:08.993122 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:08.993944 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:08.993947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:08.994153 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:08.994301 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:08.994309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.995656 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.995660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:08.996422 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:08.996436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.997033 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.997036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:08.997125 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:08.997605 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:08.997619 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:08.999086 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:09.001067 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:09.001070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:09.001072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:09.001287 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:09.001877 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:09.001881 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:09.001883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:09.002055 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:09.003173 (+ 1118us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:09.003176 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:09.003281 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 16:42:09.003369 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 16:42:09.003525 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260928 16:42:09.004750 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:08.992851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35072 (local address 127.25.71.124:43681)
0928 16:42:08.993661 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:08.993670 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:08.993687 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:42:08.993726 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:08.993728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:08.993779 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:08.993891 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:08.994729 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.995519 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.996558 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:08.996902 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:08.997364 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:08.999252 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:08.999256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:08.999258 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:08.999284 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:09.000958 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:09.001403 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:09.001407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:09.001409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:09.001505 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:09.002166 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:09.002169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:09.002171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:09.002475 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:09.004337 (+ 1862us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:09.004428 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 16:42:09.004558 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":713,"thread_start_us":429,"threads_started":1}
Sep 28 16:42:09 dist-test-slave-nqgh krb5kdc[6781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613728, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.124@KRBTEST.COM for kudu/127.25.71.125@KRBTEST.COM
Sep 28 16:42:09 dist-test-slave-nqgh krb5kdc[6781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613728, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for kudu/127.25.71.124@KRBTEST.COM
I20260928 16:42:09.009225 6937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72: Bootstrap starting.
Sep 28 16:42:09 dist-test-slave-nqgh krb5kdc[6781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:09.011199 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:09.001718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56060 (local address 127.25.71.125:38871)
0928 16:42:09.001856 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:09.001861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:09.001874 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:42:09.001906 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:09.001908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:09.001944 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:09.002005 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:09.002552 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.003306 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.004386 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.004692 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.004770 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:09.007492 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:09.007495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:09.007498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:09.007525 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:09.009178 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:09.009726 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:09.009730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:09.009732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:09.009819 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:09.010127 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:09.010130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:09.010131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:09.010400 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:09.010471 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:09.010891 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 16:42:09.011038 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:42:09.011829 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:09.001446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38871 (local address 127.0.0.1:56060)
0928 16:42:09.001665 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:09.001679 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:09.001748 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:09.002039 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:09.002042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:09.002209 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:09.002416 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:09.002424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.003437 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.003440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:09.004258 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:09.004271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.004946 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.004950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:09.005048 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:09.005629 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:09.005643 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:09.007321 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:09.009342 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:09.009346 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:09.009348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:09.009607 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:09.009935 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:09.009939 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:09.009941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:09.010028 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:09.010543 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:09.010545 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:09.010687 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 16:42:09.011476 (+ 789us) client_negotiation.cc:240] Negotiation successful
0928 16:42:09.011643 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":37}
I20260928 16:42:09.012996 6937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72: Neither blocks nor log segments found. Creating new log.
I20260928 16:42:09.013413 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:09.002587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:43681 (local address 127.0.0.1:35078)
0928 16:42:09.002781 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:09.002794 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:09.002869 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:09.003091 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:09.003094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:09.003314 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:09.003431 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:09.003438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.004650 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.004653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:09.005409 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:09.005421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.006113 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.006117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:09.006203 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:09.006903 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:09.006918 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:09.008804 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:09.011014 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:09.011019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:09.011021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:09.011350 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:09.011872 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:09.011875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:09.011877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:09.011983 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:09.012833 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:09.012837 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:09.012962 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 16:42:09.013053 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 16:42:09.013218 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260928 16:42:09.013697 7012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:42:09.013908 6937 log.cc:824] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:09.014842 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:09.002715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35078 (local address 127.25.71.124:43681)
0928 16:42:09.002880 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:09.002884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:09.002925 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:42:09.002958 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:09.002960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:09.002997 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:09.003070 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:09.003605 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.004519 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.005578 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.005974 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.006475 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:09.009022 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:09.009027 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:09.009030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:09.009067 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:09.010845 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:09.011630 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:09.011633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:09.011635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:09.011765 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:09.012108 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:09.012111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:09.012113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:09.012396 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:09.014334 (+ 1938us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:09.014420 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 16:42:09.014610 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":97}
I20260928 16:42:09.015475 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:09.002347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43580 (local address 127.25.71.126:39899)
0928 16:42:09.002481 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:09.002486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:09.002701 (+ 215us) server_negotiation.cc:407] Connection header received
0928 16:42:09.002903 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:09.002907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:09.002948 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:09.003006 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:09.003977 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.004707 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.006263 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.007118 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.007216 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:09.011444 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:09.011448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:09.011451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:09.011476 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:09.013190 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:09.013759 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:09.013762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:09.013764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:09.013846 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:09.014170 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:09.014173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:09.014175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:09.014428 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:09.014743 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:09.015182 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 16:42:09.015324 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:42:09.016458 6937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72: No bootstrap required, opened a new log
I20260928 16:42:09.019467 6862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e: Bootstrap starting.
Sep 28 16:42:09 dist-test-slave-nqgh krb5kdc[6781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613728, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.124@KRBTEST.COM for kudu/127.25.71.124@KRBTEST.COM
I20260928 16:42:09.020890 6937 raft_consensus.cc:348] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } } peers { permanent_uuid: "daa5636d6076402d88ee9ccf2e742a72" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } } peers { permanent_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } }
I20260928 16:42:09.021314 6937 raft_consensus.cc:374] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:09.021385 6937 raft_consensus.cc:729] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa5636d6076402d88ee9ccf2e742a72, State: Initialized, Role: FOLLOWER
I20260928 16:42:09.022060 6937 consensus_queue.cc:260] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } } peers { permanent_uuid: "daa5636d6076402d88ee9ccf2e742a72" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } } peers { permanent_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } }
I20260928 16:42:09.023716 6862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e: Neither blocks nor log segments found. Creating new log.
I20260928 16:42:09.023923 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:09.014396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:43681 (local address 127.0.0.1:35084)
0928 16:42:09.014861 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:09.014874 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:09.014966 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:09.015270 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:09.015272 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:09.015456 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:09.015544 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:09.015551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.016514 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.016517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:09.017227 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:09.017237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.017902 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.017905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:09.017974 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:09.018522 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:09.018536 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:09.020362 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:09.022380 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:09.022383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:09.022385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:09.022615 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:09.022947 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:09.022949 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:09.022951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:09.023017 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:09.023439 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:09.023442 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:09.023518 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 16:42:09.023603 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 16:42:09.023749 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415}
I20260928 16:42:09.024021 6937 sys_catalog.cc:564] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:42:09.024559 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:09.014673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35084 (local address 127.25.71.124:43681)
0928 16:42:09.014795 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:09.014799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:09.015041 (+ 242us) server_negotiation.cc:407] Connection header received
0928 16:42:09.015066 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:09.015069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:09.015106 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:09.015158 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:09.015659 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.016399 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.017361 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.017676 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.017772 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:09.020513 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:09.020517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:09.020519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:09.020542 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:09.022255 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:09.022786 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:09.022789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:09.022791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:09.022868 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:09.023087 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:09.023089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:09.023091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:09.023320 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:09.023408 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:09.024310 (+ 902us) server_negotiation.cc:299] Negotiation successful
0928 16:42:09.024436 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 16:42:09 dist-test-slave-nqgh krb5kdc[6781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.125@KRBTEST.COM
I20260928 16:42:09.027438 7019 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } } peers { permanent_uuid: "daa5636d6076402d88ee9ccf2e742a72" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } } peers { permanent_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } } }
I20260928 16:42:09.027727 7019 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:42:09.028868 7012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8: Bootstrap starting.
I20260928 16:42:09.030980 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:09.017938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.25.71.125:38871)
0928 16:42:09.018101 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:09.018106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:09.018754 (+ 648us) server_negotiation.cc:407] Connection header received
0928 16:42:09.018964 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:09.018968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:09.019039 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:09.019094 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:09.020091 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.021185 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.022656 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.023014 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.023098 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:09.026501 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:09.026506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:09.026509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:09.026540 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:09.028465 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:09.029091 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:09.029094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:09.029096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:09.029189 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:09.029518 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:09.029521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:09.029523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:09.029769 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:09.029959 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:09.030684 (+ 725us) server_negotiation.cc:299] Negotiation successful
0928 16:42:09.030817 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:42:09.032630 7012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8: Neither blocks nor log segments found. Creating new log.
W20260928 16:42:09.032742 7030 catalog_manager.cc:1558] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:42:09.032897 7030 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:42:09.033638 7012 log.cc:824] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:09.030390 6869 master_runner.cc:186] Error getting master registration for 127.25.71.126:39899: OK, instance_id { permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" instance_seqno: 1790613728457028 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:42:09.033916 6869 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:42:09.034281 6862 log.cc:824] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:09.036532 7012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8: No bootstrap required, opened a new log
I20260928 16:42:09.037340 6862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e: No bootstrap required, opened a new log
I20260928 16:42:09.038514 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:42:09.040457 7012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } } peers { permanent_uuid: "daa5636d6076402d88ee9ccf2e742a72" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } } peers { permanent_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } }
I20260928 16:42:09.040766 7012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:09.040856 7012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7b4201fe4545b5971ce7ab062fb5b8, State: Initialized, Role: FOLLOWER
I20260928 16:42:09.041419 7012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } } peers { permanent_uuid: "daa5636d6076402d88ee9ccf2e742a72" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } } peers { permanent_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } }
I20260928 16:42:09.042105 6862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } } peers { permanent_uuid: "daa5636d6076402d88ee9ccf2e742a72" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } } peers { permanent_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } }
I20260928 16:42:09.042376 6862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:09.042418 6862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e8c48c5784b484f9f576d95745f747e, State: Initialized, Role: FOLLOWER
I20260928 16:42:09.043001 6862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } } peers { permanent_uuid: "daa5636d6076402d88ee9ccf2e742a72" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } } peers { permanent_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } }
I20260928 16:42:09.043048 7012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:42:09.043614 7040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } } peers { permanent_uuid: "daa5636d6076402d88ee9ccf2e742a72" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } } peers { permanent_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } } }
I20260928 16:42:09.043850 7040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:42:09.045028 6862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e [sys.catalog]: configured and running, proceeding with master startup.
W20260928 16:42:09.050443 7051 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:42:09.050788 7051 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:42:09.047461 6943 master_runner.cc:186] Error getting master registration for 127.25.71.126:39899: OK, instance_id { permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" instance_seqno: 1790613728457028 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:42:09.051155 6943 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:42:09.052451 7039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } } peers { permanent_uuid: "daa5636d6076402d88ee9ccf2e742a72" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } } peers { permanent_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } } }
I20260928 16:42:09.052627 7039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 16:42:09.056175 7062 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:42:09.056396 7062 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:42:09.057237 6794 master_runner.cc:263] No leader master found from master 6e8c48c5784b484f9f576d95745f747e
I20260928 16:42:09.057317 6794 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:42:09.060602 7039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:42:09.060977 7039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } } peers { permanent_uuid: "daa5636d6076402d88ee9ccf2e742a72" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } } peers { permanent_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } }
I20260928 16:42:09.063043 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:09.041694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.25.71.124:43681)
0928 16:42:09.041887 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:09.041893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:09.053059 (+ 11166us) server_negotiation.cc:407] Connection header received
0928 16:42:09.053239 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:09.053243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:09.053301 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:09.053377 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:09.054404 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.055278 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.056578 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.056967 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.057049 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:09.059087 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:09.059091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:09.059094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:09.059128 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:09.060921 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:09.061453 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:09.061456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:09.061458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:09.061547 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:09.061853 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:09.061856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:09.061858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:09.062092 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:09.062201 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:09.062570 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 16:42:09.062762 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:42:09.063221 6836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e8c48c5784b484f9f576d95745f747e" is_pre_election: true
I20260928 16:42:09.063834 6836 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a7b4201fe4545b5971ce7ab062fb5b8 in term 0.
I20260928 16:42:09.065222 6962 leader_election.cc:302] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a7b4201fe4545b5971ce7ab062fb5b8, 6e8c48c5784b484f9f576d95745f747e; no voters:
I20260928 16:42:09.065872 7067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:42:09.066097 7067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:42:09.066206 7067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:09.066577 6911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa5636d6076402d88ee9ccf2e742a72" is_pre_election: true
I20260928 16:42:09.066989 6911 raft_consensus.cc:2454] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a7b4201fe4545b5971ce7ab062fb5b8 in term 0.
I20260928 16:42:09.067597 7039 leader_election.cc:288] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e8c48c5784b484f9f576d95745f747e (127.25.71.126:39899), daa5636d6076402d88ee9ccf2e742a72 (127.25.71.125:38871)
I20260928 16:42:09.067767 7067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } } peers { permanent_uuid: "daa5636d6076402d88ee9ccf2e742a72" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } } peers { permanent_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } }
I20260928 16:42:09.068958 7067 leader_election.cc:288] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [CANDIDATE]: Term 1 election: Requested vote from peers 6e8c48c5784b484f9f576d95745f747e (127.25.71.126:39899), daa5636d6076402d88ee9ccf2e742a72 (127.25.71.125:38871)
I20260928 16:42:09.069451 6836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e8c48c5784b484f9f576d95745f747e"
I20260928 16:42:09.069664 6836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:09.070830 6911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa5636d6076402d88ee9ccf2e742a72"
I20260928 16:42:09.071025 6911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:09.071924 6836 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a7b4201fe4545b5971ce7ab062fb5b8 in term 1.
I20260928 16:42:09.072384 6962 leader_election.cc:302] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a7b4201fe4545b5971ce7ab062fb5b8, 6e8c48c5784b484f9f576d95745f747e; no voters:
I20260928 16:42:09.072543 6911 raft_consensus.cc:2454] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a7b4201fe4545b5971ce7ab062fb5b8 in term 1.
I20260928 16:42:09.072710 7067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:42:09.072816 7067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [term 1 LEADER]: Becoming Leader. State: Replica: 2a7b4201fe4545b5971ce7ab062fb5b8, State: Running, Role: LEADER
I20260928 16:42:09.073789 7067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } } peers { permanent_uuid: "daa5636d6076402d88ee9ccf2e742a72" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } } peers { permanent_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } }
I20260928 16:42:09.078542 7039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a7b4201fe4545b5971ce7ab062fb5b8. Latest consensus state: current_term: 1 leader_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } } peers { permanent_uuid: "daa5636d6076402d88ee9ccf2e742a72" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } } peers { permanent_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } } }
I20260928 16:42:09.078979 7039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [sys.catalog]: This master's current role is: LEADER
I20260928 16:42:09.081050 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:09.053690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43588 (local address 127.25.71.126:39899)
0928 16:42:09.053824 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:09.053829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:09.058179 (+ 4350us) server_negotiation.cc:407] Connection header received
0928 16:42:09.058414 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:09.058418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:09.058478 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:09.058531 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:09.068078 (+ 9547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.068972 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.074411 (+ 5439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.074970 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.075093 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:09.077208 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:09.077213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:09.077217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:09.077256 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:09.078991 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:09.079563 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:09.079566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:09.079568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:09.079657 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:09.079960 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:09.079963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:09.079965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:09.080220 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:09.080339 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:09.080708 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 16:42:09.080863 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:42:09.082716 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:09.054308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56086 (local address 127.25.71.125:38871)
0928 16:42:09.054433 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:09.054437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:09.063890 (+ 9453us) server_negotiation.cc:407] Connection header received
0928 16:42:09.066802 (+ 2912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:09.066806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:09.066875 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:09.066932 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:09.068077 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.068975 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.070160 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:09.070460 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:09.070543 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:09.076131 (+ 5588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:09.076137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:09.076140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:09.076190 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:09.077897 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:09.078738 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:09.078743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:09.078745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:09.078861 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:09.081419 (+ 2558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:09.081425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:09.081427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:09.081789 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:09.081907 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:09.082300 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 16:42:09.082466 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:42:09.088665 7069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:42:09.092139 7069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:42:09.104202 6911 raft_consensus.cc:1261] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72 [term 1 FOLLOWER]: Refusing update from remote peer 2a7b4201fe4545b5971ce7ab062fb5b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:42:09.105422 7067 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daa5636d6076402d88ee9ccf2e742a72" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:42:09.113461 7019 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a7b4201fe4545b5971ce7ab062fb5b8. Latest consensus state: current_term: 1 leader_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } } peers { permanent_uuid: "daa5636d6076402d88ee9ccf2e742a72" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } } peers { permanent_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } } }
I20260928 16:42:09.113656 7019 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:42:09.116047 6836 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e [term 1 FOLLOWER]: Refusing update from remote peer 2a7b4201fe4545b5971ce7ab062fb5b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:42:09.117095 7067 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:42:09.123237 7067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } } peers { permanent_uuid: "daa5636d6076402d88ee9ccf2e742a72" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } } peers { permanent_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } } }
I20260928 16:42:09.123440 7067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [sys.catalog]: This master's current role is: LEADER
I20260928 16:42:09.125690 7040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a7b4201fe4545b5971ce7ab062fb5b8. Latest consensus state: current_term: 1 leader_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } } peers { permanent_uuid: "daa5636d6076402d88ee9ccf2e742a72" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } } peers { permanent_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } } }
I20260928 16:42:09.125909 7040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:42:09.126619 7019 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } } peers { permanent_uuid: "daa5636d6076402d88ee9ccf2e742a72" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } } peers { permanent_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } } }
I20260928 16:42:09.126799 7019 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:42:09.127826 7071 mvcc.cc:205] Tried to move back new op lower bound from 7334353834397331456 to 7334353834291834880. Current Snapshot: MvccSnapshot[applied={T|T < 7334353834397331456}]
I20260928 16:42:09.131157 7069 catalog_manager.cc:1347] Generated new cluster ID: 9ec2afe312464e4cac223ea944264b5b
I20260928 16:42:09.131273 7069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:42:09.146438 7039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } } peers { permanent_uuid: "daa5636d6076402d88ee9ccf2e742a72" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } } peers { permanent_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } } }
I20260928 16:42:09.149463 7039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8 [sys.catalog]: This master's current role is: LEADER
I20260928 16:42:09.150085 7073 mvcc.cc:205] Tried to move back new op lower bound from 7334353834397331456 to 7334353834291834880. Current Snapshot: MvccSnapshot[applied={T|T < 7334353834397331456}]
I20260928 16:42:09.151571 7040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c48c5784b484f9f576d95745f747e" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39899 } } peers { permanent_uuid: "daa5636d6076402d88ee9ccf2e742a72" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38871 } } peers { permanent_uuid: "2a7b4201fe4545b5971ce7ab062fb5b8" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 43681 } } }
I20260928 16:42:09.151763 7040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:42:09.165792 7069 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:42:09.167524 7069 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:42:09.194970 7069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a7b4201fe4545b5971ce7ab062fb5b8: Generated new TSK 0
I20260928 16:42:09.196372 7069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:42:10.035023 7030 catalog_manager.cc:1259] Loaded cluster ID: 9ec2afe312464e4cac223ea944264b5b
I20260928 16:42:10.035172 7030 catalog_manager.cc:1552] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72: loading cluster ID for follower catalog manager: success
I20260928 16:42:10.037359 7030 catalog_manager.cc:1574] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72: acquiring CA information for follower catalog manager: success
I20260928 16:42:10.039145 7030 catalog_manager.cc:1602] T 00000000000000000000000000000000 P daa5636d6076402d88ee9ccf2e742a72: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:42:10.058290 7062 catalog_manager.cc:1259] Loaded cluster ID: 9ec2afe312464e4cac223ea944264b5b
I20260928 16:42:10.058434 7062 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e: loading cluster ID for follower catalog manager: success
I20260928 16:42:10.060956 7062 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e: acquiring CA information for follower catalog manager: success
I20260928 16:42:10.062685 7062 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6e8c48c5784b484f9f576d95745f747e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:42:10.291388 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 6794
I20260928 16:42:10.299736 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 6869
I20260928 16:42:10.307263 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 6943
2026-09-28T16:42:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3227 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:42:10 dist-test-slave-nqgh krb5kdc[7085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:42:10 dist-test-slave-nqgh krb5kdc[7085](info): set up 2 sockets
Sep 28 16:42:10 dist-test-slave-nqgh krb5kdc[7085](info): commencing operation
krb5kdc: starting...
W20260928 16:42:11.284729 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:42:11 dist-test-slave-nqgh krb5kdc[7085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:42:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:42:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:42:11.426576 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:39279
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:38907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:39279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:42:11.549923 7098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:11.550272 7098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:11.550334 7098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:11.554733 7098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:42:11.554824 7098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:11.554854 7098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:11.554876 7098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:42:11.554898 7098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:42:11.560801 7098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:38907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:39279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:39279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:11.562088 7098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:11.563241 7098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:42:11.574056 7098 server_base.cc:1034] running on GCE node
W20260928 16:42:11.573947 7106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:11.573962 7104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:11.573961 7103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:11.575196 7098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:11.576498 7098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:11.579867 7098 hybrid_clock.cc:648] HybridClock initialized: now 1790613731579764 us; error 118 us; skew 500 ppm
Sep 28 16:42:11 dist-test-slave-nqgh krb5kdc[7085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613731, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:11.589963 7098 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:42:11.591192 7098 webserver.cc:460] Webserver started at http://127.25.71.126:33957/ using document root <none> and password file <none>
I20260928 16:42:11.591832 7098 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:11.591907 7098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:11.592123 7098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:42:11.594339 7098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "d7673051253a4206a65e11f71ace34b3"
format_stamp: "Formatted at 2026-09-28 16:42:11 on dist-test-slave-nqgh"
server_key: "83d310ba6a025a92e55cda3f8d97cd49"
server_key_iv: "00b6295a8c9c0f7584d8b08e6f2aa586"
server_key_version: "encryptionkey@0"
I20260928 16:42:11.595031 7098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "d7673051253a4206a65e11f71ace34b3"
format_stamp: "Formatted at 2026-09-28 16:42:11 on dist-test-slave-nqgh"
server_key: "83d310ba6a025a92e55cda3f8d97cd49"
server_key_iv: "00b6295a8c9c0f7584d8b08e6f2aa586"
server_key_version: "encryptionkey@0"
I20260928 16:42:11.599579 7098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:42:11.602787 7113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:11.604736 7098 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:42:11.604909 7098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "d7673051253a4206a65e11f71ace34b3"
format_stamp: "Formatted at 2026-09-28 16:42:11 on dist-test-slave-nqgh"
server_key: "83d310ba6a025a92e55cda3f8d97cd49"
server_key_iv: "00b6295a8c9c0f7584d8b08e6f2aa586"
server_key_version: "encryptionkey@0"
I20260928 16:42:11.605147 7098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:11.638324 7098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:11.642326 7098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:11.642558 7098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:11.651160 7098 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:39279
I20260928 16:42:11.651283 7165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:39279 every 8 connection(s)
I20260928 16:42:11.652257 7098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:42:11.655668 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 7098
I20260928 16:42:11.655793 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:42:11.656199 25885 external_mini_cluster.cc:1428] Setting key a9f93a90402870b8cf76f015a7bde763
I20260928 16:42:11.656672 7166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:11.666031 7166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3: Bootstrap starting.
Sep 28 16:42:11 dist-test-slave-nqgh krb5kdc[7085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:11.669899 7166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3: Neither blocks nor log segments found. Creating new log.
I20260928 16:42:11.670966 7166 log.cc:824] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:11.674297 7166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3: No bootstrap required, opened a new log
I20260928 16:42:11.675707 7169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:11.657980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52752 (local address 127.25.71.126:39279)
0928 16:42:11.658662 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:11.658689 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:11.658759 (+ 70us) server_negotiation.cc:407] Connection header received
0928 16:42:11.659767 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:11.659800 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:11.660386 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:11.660919 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:11.661786 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:11.663461 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:11.664625 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:11.665427 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:11.665646 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:11.669207 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:11.669217 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:11.669244 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:11.669304 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:11.671702 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:11.672493 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:11.672500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:11.672507 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:11.672691 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:11.673136 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:11.673139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:11.673140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:11.673708 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:11.673891 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:11.674350 (+ 459us) server_negotiation.cc:299] Negotiation successful
0928 16:42:11.674810 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":216,"threads_started":1}
I20260928 16:42:11.678295 7166 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7673051253a4206a65e11f71ace34b3" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39279 } }
I20260928 16:42:11.678694 7166 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:11.678807 7166 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7673051253a4206a65e11f71ace34b3, State: Initialized, Role: FOLLOWER
I20260928 16:42:11.679418 7166 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7673051253a4206a65e11f71ace34b3" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39279 } }
I20260928 16:42:11.679581 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:42:11.679608 7166 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:42:11.679672 7166 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:42:11.679790 7166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:11.681268 7166 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7673051253a4206a65e11f71ace34b3" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39279 } }
I20260928 16:42:11.681748 7166 leader_election.cc:302] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7673051253a4206a65e11f71ace34b3; no voters:
I20260928 16:42:11.682199 7166 leader_election.cc:288] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:42:11.682381 7171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:42:11.682801 7171 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3 [term 1 LEADER]: Becoming Leader. State: Replica: d7673051253a4206a65e11f71ace34b3, State: Running, Role: LEADER
I20260928 16:42:11.683252 7171 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7673051253a4206a65e11f71ace34b3" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39279 } }
I20260928 16:42:11.683810 7166 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:42:11.685318 7172 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7673051253a4206a65e11f71ace34b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7673051253a4206a65e11f71ace34b3" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39279 } } }
I20260928 16:42:11.685379 7173 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7673051253a4206a65e11f71ace34b3. Latest consensus state: current_term: 1 leader_uuid: "d7673051253a4206a65e11f71ace34b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7673051253a4206a65e11f71ace34b3" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39279 } } }
I20260928 16:42:11.685515 7172 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3 [sys.catalog]: This master's current role is: LEADER
I20260928 16:42:11.685542 7173 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3 [sys.catalog]: This master's current role is: LEADER
I20260928 16:42:11.686682 7180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:42:11.689845 7180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:42:11.696045 7180 catalog_manager.cc:1347] Generated new cluster ID: 54859a92d06d4a06b8263f7e5ab8bd22
I20260928 16:42:11.696166 7180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:42:11.718673 7180 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:42:11.719787 7180 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:42:11.729138 7180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7673051253a4206a65e11f71ace34b3: Generated new TSK 0
I20260928 16:42:11.729964 7180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:42:11.737443 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 7098
2026-09-28T16:42:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1427 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:42:11 dist-test-slave-nqgh krb5kdc[7193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:42:11 dist-test-slave-nqgh krb5kdc[7193](info): set up 2 sockets
Sep 28 16:42:11 dist-test-slave-nqgh krb5kdc[7193](info): commencing operation
krb5kdc: starting...
W20260928 16:42:12.732707 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:42:12 dist-test-slave-nqgh krb5kdc[7193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:42:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:42:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:42:12.871567 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:35439
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:35439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:42:13.008315 7206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:13.008741 7206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:13.008809 7206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:13.013773 7206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:42:13.013919 7206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:13.013952 7206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:13.013976 7206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:42:13.013998 7206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:42:13.021087 7206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:35439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:35439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:13.022732 7206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:13.024133 7206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:13.036373 7212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:13.036396 7211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:13.036578 7206 server_base.cc:1034] running on GCE node
W20260928 16:42:13.036391 7214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:13.037683 7206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:13.039387 7206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:13.040597 7206 hybrid_clock.cc:648] HybridClock initialized: now 1790613733040562 us; error 61 us; skew 500 ppm
Sep 28 16:42:13 dist-test-slave-nqgh krb5kdc[7193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613733, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:13.044838 7206 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:42:13.046387 7206 webserver.cc:460] Webserver started at http://127.25.71.126:43541/ using document root <none> and password file <none>
I20260928 16:42:13.047257 7206 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:13.047375 7206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:13.047662 7206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:42:13.050493 7206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "86249db880874e95aac51f00486b0647"
format_stamp: "Formatted at 2026-09-28 16:42:13 on dist-test-slave-nqgh"
server_key: "29ffac6c5f67129f2d3efec71e3eb3d0"
server_key_iv: "cd085ade24946f29250c7465da44293d"
server_key_version: "encryptionkey@0"
I20260928 16:42:13.051328 7206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "86249db880874e95aac51f00486b0647"
format_stamp: "Formatted at 2026-09-28 16:42:13 on dist-test-slave-nqgh"
server_key: "29ffac6c5f67129f2d3efec71e3eb3d0"
server_key_iv: "cd085ade24946f29250c7465da44293d"
server_key_version: "encryptionkey@0"
I20260928 16:42:13.056519 7206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:42:13.060283 7221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:13.062224 7206 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:42:13.062436 7206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "86249db880874e95aac51f00486b0647"
format_stamp: "Formatted at 2026-09-28 16:42:13 on dist-test-slave-nqgh"
server_key: "29ffac6c5f67129f2d3efec71e3eb3d0"
server_key_iv: "cd085ade24946f29250c7465da44293d"
server_key_version: "encryptionkey@0"
I20260928 16:42:13.062597 7206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:13.083243 7206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:13.087783 7206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:13.088052 7206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:13.097321 7206 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:35439
I20260928 16:42:13.097661 7273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:35439 every 8 connection(s)
I20260928 16:42:13.098757 7206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:42:13.100802 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 7206
I20260928 16:42:13.100958 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:42:13.101384 25885 external_mini_cluster.cc:1428] Setting key 03d58646754d38b50714d4ed341499fa
I20260928 16:42:13.102931 7274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:13.111905 7274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647: Bootstrap starting.
Sep 28 16:42:13 dist-test-slave-nqgh krb5kdc[7193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:13.115010 7274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647: Neither blocks nor log segments found. Creating new log.
I20260928 16:42:13.115887 7274 log.cc:824] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:13.119234 7274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647: No bootstrap required, opened a new log
I20260928 16:42:13.123682 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:13.103612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53684 (local address 127.25.71.126:35439)
0928 16:42:13.104242 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:13.104262 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:13.104313 (+ 51us) server_negotiation.cc:407] Connection header received
0928 16:42:13.105211 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:13.105235 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:13.105750 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:13.106169 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:13.107400 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:13.108798 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:13.110451 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:13.111369 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:13.111580 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:13.115077 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:13.115088 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:13.115119 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:13.115182 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:13.117696 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:13.118545 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:13.118553 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:13.118563 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:13.118886 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:13.119355 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:13.119359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:13.119361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:13.120350 (+ 989us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:13.120601 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:13.121725 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0928 16:42:13.122231 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":200,"threads_started":1}
I20260928 16:42:13.124655 7274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86249db880874e95aac51f00486b0647" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35439 } }
I20260928 16:42:13.125027 7274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:13.125092 7274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86249db880874e95aac51f00486b0647, State: Initialized, Role: FOLLOWER
I20260928 16:42:13.125603 7274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86249db880874e95aac51f00486b0647" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35439 } }
I20260928 16:42:13.125761 7274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:42:13.125824 7274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:42:13.125921 7274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:13.127245 7274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86249db880874e95aac51f00486b0647" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35439 } }
I20260928 16:42:13.127629 7274 leader_election.cc:302] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86249db880874e95aac51f00486b0647; no voters:
I20260928 16:42:13.127961 7274 leader_election.cc:288] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:42:13.128181 7279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:42:13.128492 7279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 1 LEADER]: Becoming Leader. State: Replica: 86249db880874e95aac51f00486b0647, State: Running, Role: LEADER
I20260928 16:42:13.128930 7279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86249db880874e95aac51f00486b0647" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35439 } }
I20260928 16:42:13.129451 7274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:42:13.131630 7281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86249db880874e95aac51f00486b0647. Latest consensus state: current_term: 1 leader_uuid: "86249db880874e95aac51f00486b0647" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86249db880874e95aac51f00486b0647" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35439 } } }
I20260928 16:42:13.131688 7280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86249db880874e95aac51f00486b0647" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86249db880874e95aac51f00486b0647" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35439 } } }
I20260928 16:42:13.131846 7281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [sys.catalog]: This master's current role is: LEADER
I20260928 16:42:13.131870 7280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [sys.catalog]: This master's current role is: LEADER
W20260928 16:42:13.136025 7294 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:42:13.136129 7294 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:42:13.136212 7293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:42:13.137130 7293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:42:13.145112 7293 catalog_manager.cc:1347] Generated new cluster ID: 7ddadfa61a1d4f54a7cea9833dacd8bb
I20260928 16:42:13.145256 7293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:42:13.169242 7293 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:42:13.170810 7293 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:42:13.184191 7293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647: Generated new TSK 0
I20260928 16:42:13.185266 7293 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
I20260928 16:42:13.234843 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:35439
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:42:13.382323 7300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:13.382814 7300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:13.382892 7300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:13.387730 7300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:13.387869 7300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:13.387970 7300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:42:13.395200 7300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:35439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:13.396890 7300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:13.398416 7300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:13.411093 7306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:13.411080 7305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:13.411080 7308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:13.411489 7300 server_base.cc:1034] running on GCE node
I20260928 16:42:13.412083 7300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:13.412963 7300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:13.414153 7300 hybrid_clock.cc:648] HybridClock initialized: now 1790613733414114 us; error 56 us; skew 500 ppm
Sep 28 16:42:13 dist-test-slave-nqgh krb5kdc[7193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613733, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:13.417770 7300 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:42:13.419085 7300 webserver.cc:460] Webserver started at http://127.25.71.65:34423/ using document root <none> and password file <none>
I20260928 16:42:13.419801 7300 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:13.419873 7300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:13.420092 7300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:42:13.422844 7300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "ca982d23e01e43f3ac61669de9c27699"
format_stamp: "Formatted at 2026-09-28 16:42:13 on dist-test-slave-nqgh"
server_key: "475a775c9738192e1ebba366a102a028"
server_key_iv: "ed02149879c73cfef3a6743be27feedc"
server_key_version: "encryptionkey@0"
I20260928 16:42:13.423525 7300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "ca982d23e01e43f3ac61669de9c27699"
format_stamp: "Formatted at 2026-09-28 16:42:13 on dist-test-slave-nqgh"
server_key: "475a775c9738192e1ebba366a102a028"
server_key_iv: "ed02149879c73cfef3a6743be27feedc"
server_key_version: "encryptionkey@0"
I20260928 16:42:13.428771 7300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:42:13.432744 7315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:13.434947 7300 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:42:13.435146 7300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "ca982d23e01e43f3ac61669de9c27699"
format_stamp: "Formatted at 2026-09-28 16:42:13 on dist-test-slave-nqgh"
server_key: "475a775c9738192e1ebba366a102a028"
server_key_iv: "ed02149879c73cfef3a6743be27feedc"
server_key_version: "encryptionkey@0"
I20260928 16:42:13.435309 7300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:13.448807 7300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:13.453349 7300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:13.453577 7300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:13.454366 7300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:42:13.455572 7300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:42:13.455636 7300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:13.455686 7300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:42:13.455701 7300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:13.467228 7300 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:41307
I20260928 16:42:13.467338 7428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:41307 every 8 connection(s)
I20260928 16:42:13.468533 7300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:42:13.475065 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 7300
I20260928 16:42:13.475219 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:42:13.475575 25885 external_mini_cluster.cc:1428] Setting key 6d705d76bd1233043491894c8b288a02
Sep 28 16:42:13 dist-test-slave-nqgh krb5kdc[7193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613733, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:13.484807 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:13.471084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:49229 (local address 127.25.71.126:35439)
0928 16:42:13.471292 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:13.471297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:13.472428 (+ 1131us) server_negotiation.cc:407] Connection header received
0928 16:42:13.474085 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:13.474088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:13.474154 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:13.474264 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:13.475743 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:13.476338 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:13.477655 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:13.477907 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:13.478003 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:13.481092 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:13.481095 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:13.481098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:13.481130 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:13.482547 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:13.483181 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:13.483184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:13.483185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:13.483267 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:13.483618 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:13.483621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:13.483622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:13.483832 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:13.483923 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:13.484469 (+ 546us) server_negotiation.cc:299] Negotiation successful
0928 16:42:13.484626 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 16:42:13.486083 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:13.471594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:49229)
0928 16:42:13.472228 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:13.472286 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:13.473828 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:13.474522 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:13.474530 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:13.475002 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:13.475586 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:13.475602 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:13.476535 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:13.476537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:13.477511 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:13.477523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:13.478074 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:13.478078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:13.478261 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:13.478923 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:13.478942 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:13.480868 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:13.482725 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:13.482743 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:13.482761 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:13.483076 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:13.483356 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:13.483359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:13.483361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:13.483516 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:13.483939 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:13.483947 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:13.484226 (+ 279us) client_negotiation.cc:769] Sending connection context
0928 16:42:13.484414 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 16:42:13.484791 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":150,"threads_started":1}
WARNING: no policy specified for kudu/127.25.71.66@KRBTEST.COM; defaulting to no policy
I20260928 16:42:13.487869 7429 heartbeater.cc:347] Connected to a master server at 127.25.71.126:35439
I20260928 16:42:13.488317 7429 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:13.489140 7429 heartbeater.cc:511] Master 127.25.71.126:35439 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.66@KRBTEST.COM" created.
I20260928 16:42:13.491784 7238 ts_manager.cc:194] Registered new tserver with Master: ca982d23e01e43f3ac61669de9c27699 (127.25.71.65:41307)
I20260928 16:42:13.493172 7238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:49229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
I20260928 16:42:13.512017 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:35439
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:42:13.654588 7434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:13.655157 7434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:13.655246 7434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:13.660230 7434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:13.660368 7434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:13.660490 7434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:42:13.667999 7434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:35439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:13.669607 7434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:13.671036 7434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:13.683238 7439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:13.683346 7442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:13.683313 7440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:13.683692 7434 server_base.cc:1034] running on GCE node
I20260928 16:42:13.684209 7434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:13.684974 7434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:13.686312 7434 hybrid_clock.cc:648] HybridClock initialized: now 1790613733686276 us; error 48 us; skew 500 ppm
Sep 28 16:42:13 dist-test-slave-nqgh krb5kdc[7193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613733, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:13.689718 7434 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:42:13.690891 7434 webserver.cc:460] Webserver started at http://127.25.71.66:40665/ using document root <none> and password file <none>
I20260928 16:42:13.691561 7434 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:13.691635 7434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:13.691852 7434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:42:13.694145 7434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "df42deab246142cbb05bff62c39fc80a"
format_stamp: "Formatted at 2026-09-28 16:42:13 on dist-test-slave-nqgh"
server_key: "fb2c593c842f352182f95136ff9a2104"
server_key_iv: "fb1596b8e2612b184fbf7243331869da"
server_key_version: "encryptionkey@0"
I20260928 16:42:13.694804 7434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "df42deab246142cbb05bff62c39fc80a"
format_stamp: "Formatted at 2026-09-28 16:42:13 on dist-test-slave-nqgh"
server_key: "fb2c593c842f352182f95136ff9a2104"
server_key_iv: "fb1596b8e2612b184fbf7243331869da"
server_key_version: "encryptionkey@0"
I20260928 16:42:13.699602 7434 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 16:42:13.703085 7449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:13.704957 7434 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:42:13.705134 7434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "df42deab246142cbb05bff62c39fc80a"
format_stamp: "Formatted at 2026-09-28 16:42:13 on dist-test-slave-nqgh"
server_key: "fb2c593c842f352182f95136ff9a2104"
server_key_iv: "fb1596b8e2612b184fbf7243331869da"
server_key_version: "encryptionkey@0"
I20260928 16:42:13.705271 7434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:13.721792 7434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:13.725397 7434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:13.725617 7434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:13.726265 7434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:42:13.727216 7434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:42:13.727270 7434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:13.727329 7434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:42:13.727356 7434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:13.737799 7434 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:40907
I20260928 16:42:13.737891 7562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:40907 every 8 connection(s)
I20260928 16:42:13.738946 7434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:42:13.741715 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 7434
I20260928 16:42:13.741875 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:42:13.742194 25885 external_mini_cluster.cc:1428] Setting key d1067316ae051f0ba8d37b1cd5b00b2e
Sep 28 16:42:13 dist-test-slave-nqgh krb5kdc[7193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613733, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.71.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.67@KRBTEST.COM" created.
I20260928 16:42:13.758065 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:13.741525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:39363 (local address 127.25.71.126:35439)
0928 16:42:13.741776 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:13.741782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:13.742787 (+ 1005us) server_negotiation.cc:407] Connection header received
0928 16:42:13.744627 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:13.744630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:13.744701 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:13.744776 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:13.746520 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:13.747245 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:13.748597 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:13.748918 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:13.749029 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:13.752458 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:13.752465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:13.752469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:13.752526 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:13.754521 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:13.755491 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:13.755495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:13.755497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:13.755638 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:13.756338 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:13.756342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:13.756343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:13.756675 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:13.756794 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:13.757657 (+ 863us) server_negotiation.cc:299] Negotiation successful
0928 16:42:13.757870 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 16:42:13.758875 7565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:13.741950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.66:39363)
0928 16:42:13.742557 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:13.742616 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:13.744331 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:13.745015 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:13.745023 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:13.745639 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:13.746299 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:13.746324 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:13.747450 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:13.747453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:13.748452 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:13.748465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:13.749190 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:13.749199 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:13.749425 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:13.750070 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:13.750089 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:13.752131 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:13.754864 (+ 2733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:13.754873 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:13.754890 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:13.755247 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:13.755844 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:13.755858 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:13.755860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:13.756176 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:13.756904 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:13.756911 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:13.757282 (+ 371us) client_negotiation.cc:769] Sending connection context
0928 16:42:13.757523 (+ 241us) client_negotiation.cc:240] Negotiation successful
0928 16:42:13.757863 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":163,"threads_started":1}
I20260928 16:42:13.760670 7563 heartbeater.cc:347] Connected to a master server at 127.25.71.126:35439
I20260928 16:42:13.761147 7563 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:13.761997 7563 heartbeater.cc:511] Master 127.25.71.126:35439 requested a full tablet report, sending...
I20260928 16:42:13.763720 7238 ts_manager.cc:194] Registered new tserver with Master: df42deab246142cbb05bff62c39fc80a (127.25.71.66:40907)
I20260928 16:42:13.764390 7238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:39363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
I20260928 16:42:13.778374 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:35439
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:42:13.917208 7568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:13.917623 7568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:13.917676 7568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:13.922420 7568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:13.922542 7568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:13.922614 7568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:42:13.929361 7568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:35439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:13.931005 7568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:13.932437 7568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:13.945551 7574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:13.945557 7576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:13.945557 7573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:13.945880 7568 server_base.cc:1034] running on GCE node
I20260928 16:42:13.946660 7568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:13.947606 7568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:13.948850 7568 hybrid_clock.cc:648] HybridClock initialized: now 1790613733948817 us; error 80 us; skew 500 ppm
Sep 28 16:42:13 dist-test-slave-nqgh krb5kdc[7193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613733, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:13.952958 7568 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:42:13.954420 7568 webserver.cc:460] Webserver started at http://127.25.71.67:35167/ using document root <none> and password file <none>
I20260928 16:42:13.955227 7568 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:13.955314 7568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:13.955576 7568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:42:13.958361 7568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "0cbbadcc74d249c6aa3a29d6010cb055"
format_stamp: "Formatted at 2026-09-28 16:42:13 on dist-test-slave-nqgh"
server_key: "559cff2ce6b3b876694aca66e39e1f44"
server_key_iv: "15e1a9bf92617c5c37cc612ab5e017f4"
server_key_version: "encryptionkey@0"
I20260928 16:42:13.959091 7568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "0cbbadcc74d249c6aa3a29d6010cb055"
format_stamp: "Formatted at 2026-09-28 16:42:13 on dist-test-slave-nqgh"
server_key: "559cff2ce6b3b876694aca66e39e1f44"
server_key_iv: "15e1a9bf92617c5c37cc612ab5e017f4"
server_key_version: "encryptionkey@0"
I20260928 16:42:13.964421 7568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:42:13.968412 7583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:13.970683 7568 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:42:13.970908 7568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "0cbbadcc74d249c6aa3a29d6010cb055"
format_stamp: "Formatted at 2026-09-28 16:42:13 on dist-test-slave-nqgh"
server_key: "559cff2ce6b3b876694aca66e39e1f44"
server_key_iv: "15e1a9bf92617c5c37cc612ab5e017f4"
server_key_version: "encryptionkey@0"
I20260928 16:42:13.971089 7568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:13.989596 7568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:13.993180 7568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:13.993398 7568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:13.994047 7568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:42:13.995062 7568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:42:13.995116 7568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:13.995174 7568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:42:13.995203 7568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:14.005180 7568 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:34275
I20260928 16:42:14.005205 7697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:34275 every 8 connection(s)
I20260928 16:42:14.006232 7568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:42:14.007743 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 7568
I20260928 16:42:14.007905 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:42:14.008302 25885 external_mini_cluster.cc:1428] Setting key 7fb6d506cc99925c4360e04cc9b4356e
Sep 28 16:42:14 dist-test-slave-nqgh krb5kdc[7193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613733, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:14.025120 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.009108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:48575 (local address 127.25.71.126:35439)
0928 16:42:14.009319 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:14.009325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:14.010617 (+ 1292us) server_negotiation.cc:407] Connection header received
0928 16:42:14.012525 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:14.012528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:14.012588 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:14.012694 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:14.014352 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.014997 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.016413 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.016675 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.016795 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:14.020339 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:14.020343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:14.020345 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:14.020376 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:14.021827 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:14.022658 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:14.022664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:14.022666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:14.022853 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:14.023403 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:14.023406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:14.023408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:14.023656 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:14.023766 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:14.024796 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0928 16:42:14.024978 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:42:14.026360 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.009635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.67:48575)
0928 16:42:14.010388 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:14.010466 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:14.012236 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:14.012912 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:14.012922 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:14.013429 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:14.014137 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:14.014157 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.015165 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.015167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:14.016247 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:14.016258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.016978 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.016981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:14.017215 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:14.017966 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:14.017987 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:14.020094 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:14.021991 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:14.022000 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:14.022022 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:14.022393 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:14.023003 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:14.023006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:14.023008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:14.023244 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:14.023849 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:14.023861 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:14.024429 (+ 568us) client_negotiation.cc:769] Sending connection context
0928 16:42:14.024739 (+ 310us) client_negotiation.cc:240] Negotiation successful
0928 16:42:14.025214 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":172,"threads_started":1}
I20260928 16:42:14.028000 7698 heartbeater.cc:347] Connected to a master server at 127.25.71.126:35439
I20260928 16:42:14.028440 7698 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:14.029175 7698 heartbeater.cc:511] Master 127.25.71.126:35439 requested a full tablet report, sending...
I20260928 16:42:14.030850 7238 ts_manager.cc:194] Registered new tserver with Master: 0cbbadcc74d249c6aa3a29d6010cb055 (127.25.71.67:34275)
I20260928 16:42:14.031507 7238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:48575
I20260928 16:42:14.037607 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:42:14.049154 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.040022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53698 (local address 127.25.71.126:35439)
0928 16:42:14.040198 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:14.040203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:14.040285 (+ 82us) server_negotiation.cc:407] Connection header received
0928 16:42:14.040407 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:14.040409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:14.040467 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:14.040573 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:14.041477 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.042134 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.043243 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.043513 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.043613 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:14.045254 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:14.045257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:14.045260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:14.045291 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:14.046995 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:14.047533 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:14.047537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:14.047539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:14.047644 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:14.047930 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:14.047933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:14.047934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:14.048314 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:14.048447 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:14.048765 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 16:42:14.048914 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 16:42:14.053078 7238 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:42:14.056579 7238 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:42:14.075896 7709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.068812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:40907 (local address 127.0.0.1:58530)
0928 16:42:14.069229 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:14.069284 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:14.069436 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:14.071150 (+ 1714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:14.071154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:14.071176 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:14.071478 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:14.071490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.072513 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.072516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:14.073746 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:14.073756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.075229 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.075232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:14.075369 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:14.075414 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 16:42:14.075538 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 16:42:14.075699 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":131,"thread_start_us":115,"threads_started":1}
I20260928 16:42:14.076522 7713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.069751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58530 (local address 127.25.71.66:40907)
0928 16:42:14.070706 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:14.070717 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:14.070739 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:42:14.070817 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:14.070822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:14.070997 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:14.071377 (+ 380us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:14.071585 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.072381 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.073900 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.075083 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.075219 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:14.075641 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:14.075705 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 16:42:14.075814 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":209,"threads_started":1}
I20260928 16:42:14.076663 7710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.068837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:34275 (local address 127.0.0.1:60206)
0928 16:42:14.069766 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:14.069784 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:14.069919 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:14.071086 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:14.071095 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:14.071128 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:14.071457 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:14.071469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.073259 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.073263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:14.074815 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:14.074829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.076094 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.076098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:14.076198 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:14.076220 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 16:42:14.076266 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 16:42:14.076461 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"mutex_wait_us":125,"thread_start_us":265,"threads_started":1}
I20260928 16:42:14.076675 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.069778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:41307 (local address 127.0.0.1:51116)
0928 16:42:14.070281 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:14.070298 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:14.070410 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:14.071529 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:14.071532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:14.071554 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:14.071684 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:14.071692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.073373 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.073377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:14.074681 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:14.074696 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.076042 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.076048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:14.076159 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:14.076203 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 16:42:14.076266 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 16:42:14.076406 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":42,"thread_start_us":195,"threads_started":1}
I20260928 16:42:14.077059 7711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.069237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60206 (local address 127.25.71.67:34275)
0928 16:42:14.069947 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:14.069961 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:14.069992 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:42:14.070085 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:14.070090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:14.070819 (+ 729us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:14.071050 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:14.071648 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.073041 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.074990 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.075952 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.076365 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:14.076582 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:14.076652 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 16:42:14.076883 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":147,"threads_started":1}
I20260928 16:42:14.077807 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.070336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.25.71.65:41307)
0928 16:42:14.070776 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:14.070786 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:14.070818 (+ 32us) server_negotiation.cc:407] Connection header received
0928 16:42:14.071040 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:14.071049 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:14.071318 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:14.071878 (+ 560us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:14.072072 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.073233 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.074821 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.075830 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.077258 (+ 1428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:14.077449 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:14.077509 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 16:42:14.077630 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":123,"threads_started":1}
I20260928 16:42:14.080303 7497 tablet_service.cc:1467] Processing CreateTablet for tablet f2221ce185f64624b9a38e8f53356407 (DEFAULT_TABLE table=test-table [id=bfe5f5990a084fd0b08953dc2e0f9c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:42:14.080765 7363 tablet_service.cc:1467] Processing CreateTablet for tablet f2221ce185f64624b9a38e8f53356407 (DEFAULT_TABLE table=test-table [id=bfe5f5990a084fd0b08953dc2e0f9c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:42:14.080768 7631 tablet_service.cc:1467] Processing CreateTablet for tablet f2221ce185f64624b9a38e8f53356407 (DEFAULT_TABLE table=test-table [id=bfe5f5990a084fd0b08953dc2e0f9c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:42:14.081667 7497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2221ce185f64624b9a38e8f53356407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:14.082099 7363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2221ce185f64624b9a38e8f53356407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:14.082106 7631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2221ce185f64624b9a38e8f53356407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:14.089107 7715 tablet_bootstrap.cc:492] T f2221ce185f64624b9a38e8f53356407 P df42deab246142cbb05bff62c39fc80a: Bootstrap starting.
I20260928 16:42:14.092021 7715 tablet_bootstrap.cc:654] T f2221ce185f64624b9a38e8f53356407 P df42deab246142cbb05bff62c39fc80a: Neither blocks nor log segments found. Creating new log.
I20260928 16:42:14.092912 7715 log.cc:824] T f2221ce185f64624b9a38e8f53356407 P df42deab246142cbb05bff62c39fc80a: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:14.093791 7717 tablet_bootstrap.cc:492] T f2221ce185f64624b9a38e8f53356407 P 0cbbadcc74d249c6aa3a29d6010cb055: Bootstrap starting.
I20260928 16:42:14.093925 7716 tablet_bootstrap.cc:492] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699: Bootstrap starting.
I20260928 16:42:14.095363 7715 tablet_bootstrap.cc:492] T f2221ce185f64624b9a38e8f53356407 P df42deab246142cbb05bff62c39fc80a: No bootstrap required, opened a new log
I20260928 16:42:14.095585 7715 ts_tablet_manager.cc:1397] T f2221ce185f64624b9a38e8f53356407 P df42deab246142cbb05bff62c39fc80a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 16:42:14.096598 7716 tablet_bootstrap.cc:654] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699: Neither blocks nor log segments found. Creating new log.
I20260928 16:42:14.097148 7717 tablet_bootstrap.cc:654] T f2221ce185f64624b9a38e8f53356407 P 0cbbadcc74d249c6aa3a29d6010cb055: Neither blocks nor log segments found. Creating new log.
I20260928 16:42:14.097522 7716 log.cc:824] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:14.098165 7717 log.cc:824] T f2221ce185f64624b9a38e8f53356407 P 0cbbadcc74d249c6aa3a29d6010cb055: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:14.099722 7715 raft_consensus.cc:348] T f2221ce185f64624b9a38e8f53356407 P df42deab246142cbb05bff62c39fc80a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:14.100042 7716 tablet_bootstrap.cc:492] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699: No bootstrap required, opened a new log
I20260928 16:42:14.100042 7715 raft_consensus.cc:374] T f2221ce185f64624b9a38e8f53356407 P df42deab246142cbb05bff62c39fc80a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:14.100097 7715 raft_consensus.cc:729] T f2221ce185f64624b9a38e8f53356407 P df42deab246142cbb05bff62c39fc80a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df42deab246142cbb05bff62c39fc80a, State: Initialized, Role: FOLLOWER
I20260928 16:42:14.100255 7716 ts_tablet_manager.cc:1397] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 16:42:14.100737 7715 consensus_queue.cc:260] T f2221ce185f64624b9a38e8f53356407 P df42deab246142cbb05bff62c39fc80a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:14.101990 7715 ts_tablet_manager.cc:1428] T f2221ce185f64624b9a38e8f53356407 P df42deab246142cbb05bff62c39fc80a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 16:42:14.102442 7563 heartbeater.cc:503] Master 127.25.71.126:35439 was elected leader, sending a full tablet report...
I20260928 16:42:14.104723 7717 tablet_bootstrap.cc:492] T f2221ce185f64624b9a38e8f53356407 P 0cbbadcc74d249c6aa3a29d6010cb055: No bootstrap required, opened a new log
I20260928 16:42:14.104984 7717 ts_tablet_manager.cc:1397] T f2221ce185f64624b9a38e8f53356407 P 0cbbadcc74d249c6aa3a29d6010cb055: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260928 16:42:14.105125 7716 raft_consensus.cc:348] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:14.105450 7716 raft_consensus.cc:374] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:14.105518 7716 raft_consensus.cc:729] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca982d23e01e43f3ac61669de9c27699, State: Initialized, Role: FOLLOWER
I20260928 16:42:14.105997 7716 consensus_queue.cc:260] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:14.107133 7716 ts_tablet_manager.cc:1428] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 16:42:14.107554 7429 heartbeater.cc:503] Master 127.25.71.126:35439 was elected leader, sending a full tablet report...
I20260928 16:42:14.109161 7717 raft_consensus.cc:348] T f2221ce185f64624b9a38e8f53356407 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:14.109617 7717 raft_consensus.cc:374] T f2221ce185f64624b9a38e8f53356407 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:14.109694 7717 raft_consensus.cc:729] T f2221ce185f64624b9a38e8f53356407 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cbbadcc74d249c6aa3a29d6010cb055, State: Initialized, Role: FOLLOWER
I20260928 16:42:14.110432 7717 consensus_queue.cc:260] T f2221ce185f64624b9a38e8f53356407 P 0cbbadcc74d249c6aa3a29d6010cb055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:14.112170 7698 heartbeater.cc:503] Master 127.25.71.126:35439 was elected leader, sending a full tablet report...
I20260928 16:42:14.112727 7717 ts_tablet_manager.cc:1428] T f2221ce185f64624b9a38e8f53356407 P 0cbbadcc74d249c6aa3a29d6010cb055: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260928 16:42:14.116748 7722 raft_consensus.cc:482] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:42:14.117007 7722 raft_consensus.cc:504] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:14.118263 7722 leader_election.cc:288] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cbbadcc74d249c6aa3a29d6010cb055 (127.25.71.67:34275), df42deab246142cbb05bff62c39fc80a (127.25.71.66:40907)
I20260928 16:42:14.123340 7711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.119130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:52975 (local address 127.25.71.67:34275)
0928 16:42:14.119267 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:14.119271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:14.119380 (+ 109us) server_negotiation.cc:407] Connection header received
0928 16:42:14.119540 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:14.119543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:14.119604 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:14.119682 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:14.120004 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.120576 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.122073 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.122551 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.122712 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:14.122782 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:14.123064 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 16:42:14.123167 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:42:14.123370 7713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.118833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:37027 (local address 127.25.71.66:40907)
0928 16:42:14.119047 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:14.119053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:14.119591 (+ 538us) server_negotiation.cc:407] Connection header received
0928 16:42:14.119779 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:14.119782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:14.119835 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:14.119901 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:14.120222 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.120838 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.122205 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.122720 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.122868 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:14.122919 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:14.123124 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 16:42:14.123215 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 16:42:14.123405 7724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.118745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:34275 (local address 127.25.71.65:52975)
0928 16:42:14.119255 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:14.119271 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:14.119420 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:14.119710 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:14.119714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:14.119736 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:14.119896 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:14.119903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.120722 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.120726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:14.121924 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:14.121937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.122715 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.122718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:14.122855 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:14.122943 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 16:42:14.123021 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 16:42:14.123109 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":33,"thread_start_us":164,"threads_started":1}
I20260928 16:42:14.123395 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.118745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:40907 (local address 127.25.71.65:37027)
0928 16:42:14.119470 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:14.119484 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:14.119625 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:14.119944 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:14.119948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:14.119967 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:14.120112 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:14.120118 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.120968 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.120972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:14.122047 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:14.122060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.122867 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.122871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:14.122987 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:14.123011 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:42:14.123095 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 16:42:14.123183 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":68,"thread_start_us":168,"threads_started":1}
I20260928 16:42:14.124396 7651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2221ce185f64624b9a38e8f53356407" candidate_uuid: "ca982d23e01e43f3ac61669de9c27699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" is_pre_election: true
I20260928 16:42:14.124678 7517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2221ce185f64624b9a38e8f53356407" candidate_uuid: "ca982d23e01e43f3ac61669de9c27699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df42deab246142cbb05bff62c39fc80a" is_pre_election: true
I20260928 16:42:14.124789 7651 raft_consensus.cc:2454] T f2221ce185f64624b9a38e8f53356407 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca982d23e01e43f3ac61669de9c27699 in term 0.
I20260928 16:42:14.124958 7517 raft_consensus.cc:2454] T f2221ce185f64624b9a38e8f53356407 P df42deab246142cbb05bff62c39fc80a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca982d23e01e43f3ac61669de9c27699 in term 0.
I20260928 16:42:14.125447 7317 leader_election.cc:302] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cbbadcc74d249c6aa3a29d6010cb055, ca982d23e01e43f3ac61669de9c27699; no voters:
I20260928 16:42:14.125846 7722 raft_consensus.cc:2781] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:42:14.125959 7722 raft_consensus.cc:482] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:42:14.126024 7722 raft_consensus.cc:3037] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:14.127565 7722 raft_consensus.cc:504] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:14.128067 7722 leader_election.cc:288] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 1 election: Requested vote from peers 0cbbadcc74d249c6aa3a29d6010cb055 (127.25.71.67:34275), df42deab246142cbb05bff62c39fc80a (127.25.71.66:40907)
I20260928 16:42:14.128729 7517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2221ce185f64624b9a38e8f53356407" candidate_uuid: "ca982d23e01e43f3ac61669de9c27699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df42deab246142cbb05bff62c39fc80a"
I20260928 16:42:14.128679 7651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2221ce185f64624b9a38e8f53356407" candidate_uuid: "ca982d23e01e43f3ac61669de9c27699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cbbadcc74d249c6aa3a29d6010cb055"
I20260928 16:42:14.128908 7517 raft_consensus.cc:3037] T f2221ce185f64624b9a38e8f53356407 P df42deab246142cbb05bff62c39fc80a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:14.128935 7651 raft_consensus.cc:3037] T f2221ce185f64624b9a38e8f53356407 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:14.130450 7651 raft_consensus.cc:2454] T f2221ce185f64624b9a38e8f53356407 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca982d23e01e43f3ac61669de9c27699 in term 1.
I20260928 16:42:14.130450 7517 raft_consensus.cc:2454] T f2221ce185f64624b9a38e8f53356407 P df42deab246142cbb05bff62c39fc80a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca982d23e01e43f3ac61669de9c27699 in term 1.
I20260928 16:42:14.131001 7317 leader_election.cc:302] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0cbbadcc74d249c6aa3a29d6010cb055, ca982d23e01e43f3ac61669de9c27699, df42deab246142cbb05bff62c39fc80a; no voters:
I20260928 16:42:14.131314 7722 raft_consensus.cc:2781] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:42:14.131665 7722 raft_consensus.cc:686] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [term 1 LEADER]: Becoming Leader. State: Replica: ca982d23e01e43f3ac61669de9c27699, State: Running, Role: LEADER
I20260928 16:42:14.132133 7722 consensus_queue.cc:237] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:14.136404 7238 catalog_manager.cc:5495] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca982d23e01e43f3ac61669de9c27699 (127.25.71.65). New cstate: current_term: 1 leader_uuid: "ca982d23e01e43f3ac61669de9c27699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } health_report { overall_health: HEALTHY } } }
I20260928 16:42:14.162932 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.158839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51128 (local address 127.25.71.65:41307)
0928 16:42:14.159054 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:14.159060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:14.159078 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:42:14.159188 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:14.159191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:14.159260 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:14.159376 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:42:14.159666 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.160241 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.161157 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.161381 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.161470 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:14.161826 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:42:14.162431 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:42:14.162548 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:14.162656 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 16:42:14.162751 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 16:42:14.171406 7517 raft_consensus.cc:1261] T f2221ce185f64624b9a38e8f53356407 P df42deab246142cbb05bff62c39fc80a [term 1 FOLLOWER]: Refusing update from remote peer ca982d23e01e43f3ac61669de9c27699: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:42:14.171456 7651 raft_consensus.cc:1261] T f2221ce185f64624b9a38e8f53356407 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 1 FOLLOWER]: Refusing update from remote peer ca982d23e01e43f3ac61669de9c27699: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:42:14.172392 7726 consensus_queue.cc:1035] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:42:14.172698 7722 consensus_queue.cc:1035] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:42:14.181715 7729 mvcc.cc:205] Tried to move back new op lower bound from 7334353855155159040 to 7334353855009116160. Current Snapshot: MvccSnapshot[applied={T|T < 7334353855155159040}]
I20260928 16:42:14.184296 7731 mvcc.cc:205] Tried to move back new op lower bound from 7334353855155159040 to 7334353855009116160. Current Snapshot: MvccSnapshot[applied={T|T < 7334353855155159040}]
I20260928 16:42:14.193918 7237 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:42:14.194214 7237 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:42:14.197664 7237 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647: Sending DeleteTablet for 3 replicas of tablet f2221ce185f64624b9a38e8f53356407
I20260928 16:42:14.198917 7497 tablet_service.cc:1514] Processing DeleteTablet for tablet f2221ce185f64624b9a38e8f53356407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:42:14 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:58530
I20260928 16:42:14.199055 7363 tablet_service.cc:1514] Processing DeleteTablet for tablet f2221ce185f64624b9a38e8f53356407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:42:14 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:51116
I20260928 16:42:14.199055 7631 tablet_service.cc:1514] Processing DeleteTablet for tablet f2221ce185f64624b9a38e8f53356407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:42:14 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:60206
I20260928 16:42:14.199755 7739 tablet_replica.cc:330] T f2221ce185f64624b9a38e8f53356407 P 0cbbadcc74d249c6aa3a29d6010cb055: stopping tablet replica
I20260928 16:42:14.200098 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 7300
I20260928 16:42:14.200186 7739 raft_consensus.cc:2229] T f2221ce185f64624b9a38e8f53356407 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:42:14.201507 7739 raft_consensus.cc:2258] T f2221ce185f64624b9a38e8f53356407 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:42:14.202505 7738 tablet_replica.cc:330] T f2221ce185f64624b9a38e8f53356407 P df42deab246142cbb05bff62c39fc80a: stopping tablet replica
I20260928 16:42:14.203011 7738 raft_consensus.cc:2229] T f2221ce185f64624b9a38e8f53356407 P df42deab246142cbb05bff62c39fc80a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:42:14.203408 7738 raft_consensus.cc:2258] T f2221ce185f64624b9a38e8f53356407 P df42deab246142cbb05bff62c39fc80a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:42:14.204373 7739 ts_tablet_manager.cc:1905] T f2221ce185f64624b9a38e8f53356407 P 0cbbadcc74d249c6aa3a29d6010cb055: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:42:14.205010 7738 ts_tablet_manager.cc:1905] T f2221ce185f64624b9a38e8f53356407 P df42deab246142cbb05bff62c39fc80a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:42:14.207686 7739 ts_tablet_manager.cc:1918] T f2221ce185f64624b9a38e8f53356407 P 0cbbadcc74d249c6aa3a29d6010cb055: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:42:14.207841 7739 log.cc:1196] T f2221ce185f64624b9a38e8f53356407 P 0cbbadcc74d249c6aa3a29d6010cb055: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/f2221ce185f64624b9a38e8f53356407
I20260928 16:42:14.207896 7738 ts_tablet_manager.cc:1918] T f2221ce185f64624b9a38e8f53356407 P df42deab246142cbb05bff62c39fc80a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:42:14.208040 7738 log.cc:1196] T f2221ce185f64624b9a38e8f53356407 P df42deab246142cbb05bff62c39fc80a: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/f2221ce185f64624b9a38e8f53356407
I20260928 16:42:14.208225 7739 ts_tablet_manager.cc:1939] T f2221ce185f64624b9a38e8f53356407 P 0cbbadcc74d249c6aa3a29d6010cb055: Deleting consensus metadata
I20260928 16:42:14.208355 7738 ts_tablet_manager.cc:1939] T f2221ce185f64624b9a38e8f53356407 P df42deab246142cbb05bff62c39fc80a: Deleting consensus metadata
I20260928 16:42:14.209578 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 7434
W20260928 16:42:14.209620 7224 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.71.65:41307 (error 108)
I20260928 16:42:14.210314 7223 catalog_manager.cc:4841] TS 0cbbadcc74d249c6aa3a29d6010cb055 (127.25.71.67:34275): tablet f2221ce185f64624b9a38e8f53356407 (table test-table [id=bfe5f5990a084fd0b08953dc2e0f9c0d]) successfully deleted
I20260928 16:42:14.211313 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.210847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:41307 (local address 127.0.0.1:51138)
0928 16:42:14.211143 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:14.211215 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.65:41307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
W20260928 16:42:14.211498 7224 catalog_manager.cc:4584] TS ca982d23e01e43f3ac61669de9c27699 (127.25.71.65:41307): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f2221ce185f64624b9a38e8f53356407: Network error: Client connection negotiation failed: client connection to 127.25.71.65:41307: connect: Connection refused (error 111)
I20260928 16:42:14.216939 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 7568
I20260928 16:42:14.219403 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.218951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:40907 (local address 127.0.0.1:58544)
0928 16:42:14.219216 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:14.219279 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.66:40907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260928 16:42:14.222877 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 7206
I20260928 16:42:14.229792 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:35439
--webserver_interface=127.25.71.126
--webserver_port=43541
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:35439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:42:14.358528 7742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:14.358911 7742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:14.358958 7742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:14.363368 7742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:42:14.363462 7742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:14.363482 7742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:14.363500 7742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:42:14.363514 7742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:42:14.369530 7742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:35439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:35439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=43541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:14.370872 7742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:14.372041 7742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:14.388265 7751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:14.388270 7749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:14.388270 7748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:14.388507 7742 server_base.cc:1034] running on GCE node
I20260928 16:42:14.389588 7742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:14.390976 7742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:14.392235 7742 hybrid_clock.cc:648] HybridClock initialized: now 1790613734392207 us; error 62 us; skew 500 ppm
Sep 28 16:42:14 dist-test-slave-nqgh krb5kdc[7193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613734, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:14.395399 7742 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:42:14.396585 7742 webserver.cc:460] Webserver started at http://127.25.71.126:43541/ using document root <none> and password file <none>
I20260928 16:42:14.397310 7742 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:14.397416 7742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:14.402454 7742 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260928 16:42:14.406898 7758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:14.409134 7742 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:42:14.409324 7742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "86249db880874e95aac51f00486b0647"
format_stamp: "Formatted at 2026-09-28 16:42:13 on dist-test-slave-nqgh"
server_key: "29ffac6c5f67129f2d3efec71e3eb3d0"
server_key_iv: "cd085ade24946f29250c7465da44293d"
server_key_version: "encryptionkey@0"
I20260928 16:42:14.410072 7742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:14.435036 7742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:14.438673 7742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:14.438886 7742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:14.447654 7742 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:35439
I20260928 16:42:14.448041 7810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:35439 every 8 connection(s)
I20260928 16:42:14.449038 7742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:42:14.455493 7811 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:42:14.457983 7811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647: Bootstrap starting.
I20260928 16:42:14.459352 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 7742
I20260928 16:42:14.459908 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:41307
--local_ip_for_outbound_sockets=127.25.71.65
--tserver_master_addrs=127.25.71.126:35439
--webserver_port=34423
--webserver_interface=127.25.71.65
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:42:14.479936 7811 log.cc:824] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:14.491698 7811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:42:14.492146 7811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647: Bootstrap complete.
I20260928 16:42:14.496599 7811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86249db880874e95aac51f00486b0647" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35439 } }
I20260928 16:42:14.497079 7811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86249db880874e95aac51f00486b0647, State: Initialized, Role: FOLLOWER
I20260928 16:42:14.497606 7811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86249db880874e95aac51f00486b0647" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35439 } }
I20260928 16:42:14.497745 7811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:42:14.497834 7811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:42:14.497936 7811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:42:14.500174 7811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86249db880874e95aac51f00486b0647" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35439 } }
I20260928 16:42:14.500538 7811 leader_election.cc:302] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86249db880874e95aac51f00486b0647; no voters:
I20260928 16:42:14.500880 7811 leader_election.cc:288] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 16:42:14.501020 7815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:42:14.501281 7815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 2 LEADER]: Becoming Leader. State: Replica: 86249db880874e95aac51f00486b0647, State: Running, Role: LEADER
I20260928 16:42:14.501655 7815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86249db880874e95aac51f00486b0647" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35439 } }
I20260928 16:42:14.502384 7811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:42:14.503562 7816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "86249db880874e95aac51f00486b0647" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86249db880874e95aac51f00486b0647" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35439 } } }
I20260928 16:42:14.503718 7816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [sys.catalog]: This master's current role is: LEADER
I20260928 16:42:14.503594 7817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86249db880874e95aac51f00486b0647. Latest consensus state: current_term: 2 leader_uuid: "86249db880874e95aac51f00486b0647" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86249db880874e95aac51f00486b0647" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35439 } } }
I20260928 16:42:14.503800 7817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [sys.catalog]: This master's current role is: LEADER
I20260928 16:42:14.504438 7824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:42:14.509208 7824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:42:14.509805 7824 catalog_manager.cc:1259] Loaded cluster ID: 7ddadfa61a1d4f54a7cea9833dacd8bb
I20260928 16:42:14.509864 7824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:42:14.511125 7824 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 16:42:14.513974 7824 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 16:42:14.521705 7824 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:42:14.522876 7824 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647: Loaded TSK: 0
I20260928 16:42:14.524307 7824 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:42:14.603284 7813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:14.603704 7813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:14.603777 7813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:14.608711 7813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:14.608815 7813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:14.608916 7813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:42:14.615633 7813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:41307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=34423
--tserver_master_addrs=127.25.71.126:35439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:14.617341 7813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:14.618774 7813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:14.631981 7838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:14.632148 7841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:14.631973 7839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:14.632738 7813 server_base.cc:1034] running on GCE node
I20260928 16:42:14.633356 7813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:14.634382 7813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:14.635648 7813 hybrid_clock.cc:648] HybridClock initialized: now 1790613734635603 us; error 66 us; skew 500 ppm
Sep 28 16:42:14 dist-test-slave-nqgh krb5kdc[7193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613734, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:14.639748 7813 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:42:14.641180 7813 webserver.cc:460] Webserver started at http://127.25.71.65:34423/ using document root <none> and password file <none>
I20260928 16:42:14.641985 7813 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:14.642100 7813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:14.648196 7813 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:42:14.652084 7848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:14.654284 7813 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:42:14.654513 7813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "ca982d23e01e43f3ac61669de9c27699"
format_stamp: "Formatted at 2026-09-28 16:42:13 on dist-test-slave-nqgh"
server_key: "475a775c9738192e1ebba366a102a028"
server_key_iv: "ed02149879c73cfef3a6743be27feedc"
server_key_version: "encryptionkey@0"
I20260928 16:42:14.655508 7813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:14.680259 7813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:14.684371 7813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:14.684567 7813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:14.685256 7813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:42:14.686825 7855 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 16:42:14.692974 7813 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 16:42:14.693109 7813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260928 16:42:14.693193 7813 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 16:42:14.695420 7813 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 16:42:14.695485 7813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260928 16:42:14.695946 7855 tablet_bootstrap.cc:492] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699: Bootstrap starting.
I20260928 16:42:14.706319 7813 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:41307
I20260928 16:42:14.706600 7962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:41307 every 8 connection(s)
I20260928 16:42:14.707505 7813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:42:14.708428 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 7813
I20260928 16:42:14.708880 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:40907
--local_ip_for_outbound_sockets=127.25.71.66
--tserver_master_addrs=127.25.71.126:35439
--webserver_port=40665
--webserver_interface=127.25.71.66
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:42:14 dist-test-slave-nqgh krb5kdc[7193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613734, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:14.721251 7855 log.cc:824] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:14.727231 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.710001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:36805)
0928 16:42:14.711006 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:14.711074 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:14.712182 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:14.713534 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:14.713544 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:14.714135 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:14.714845 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:14.714869 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.716480 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.716484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:14.717455 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:14.717468 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.718268 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.718272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:14.718474 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:14.719186 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:14.719206 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:14.720851 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:14.723418 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:14.723425 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:14.723440 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:14.723732 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:14.724109 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:14.724113 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:14.724115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:14.724263 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:14.724971 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:14.724978 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:14.725347 (+ 369us) client_negotiation.cc:769] Sending connection context
0928 16:42:14.725537 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 16:42:14.725981 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":188,"threads_started":1}
I20260928 16:42:14.727205 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.710030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:36805 (local address 127.25.71.126:35439)
0928 16:42:14.711467 (+ 1437us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:14.711488 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:14.711546 (+ 58us) server_negotiation.cc:407] Connection header received
0928 16:42:14.712679 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:14.712690 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:14.713060 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:14.713415 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:14.715050 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.716293 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.717595 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.718106 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.718287 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:14.721076 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:14.721083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:14.721098 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:14.721131 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:14.723283 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:14.723867 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:14.723874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:14.723878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:14.724014 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:14.724364 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:14.724367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:14.724368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:14.724848 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:14.724985 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:14.725709 (+ 724us) server_negotiation.cc:299] Negotiation successful
0928 16:42:14.726156 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1283,"thread_start_us":169,"threads_started":1}
I20260928 16:42:14.730403 7963 heartbeater.cc:347] Connected to a master server at 127.25.71.126:35439
I20260928 16:42:14.730767 7963 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:14.731385 7855 tablet_bootstrap.cc:492] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:42:14.731451 7963 heartbeater.cc:511] Master 127.25.71.126:35439 requested a full tablet report, sending...
I20260928 16:42:14.731854 7855 tablet_bootstrap.cc:492] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699: Bootstrap complete.
I20260928 16:42:14.733234 7855 ts_tablet_manager.cc:1397] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699: Time spent bootstrapping tablet: real 0.038s user 0.025s sys 0.008s
I20260928 16:42:14.734612 7775 ts_manager.cc:194] Registered new tserver with Master: ca982d23e01e43f3ac61669de9c27699 (127.25.71.65:41307)
I20260928 16:42:14.736419 7855 raft_consensus.cc:348] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:14.737007 7855 raft_consensus.cc:729] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca982d23e01e43f3ac61669de9c27699, State: Initialized, Role: FOLLOWER
I20260928 16:42:14.737574 7855 consensus_queue.cc:260] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:14.738921 7855 ts_tablet_manager.cc:1428] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260928 16:42:14.740614 7775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:36805
I20260928 16:42:14.744169 7963 heartbeater.cc:503] Master 127.25.71.126:35439 was elected leader, sending a full tablet report...
Sep 28 16:42:14 dist-test-slave-nqgh krb5kdc[7193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613734, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for kudu/127.25.71.65@KRBTEST.COM
I20260928 16:42:14.752648 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.739338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39364 (local address 127.25.71.65:41307)
0928 16:42:14.739753 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:14.739767 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:14.739802 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:42:14.739907 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:14.739913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:14.740144 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:14.740318 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:14.741232 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.742363 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.743842 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.744412 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.744550 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:14.747999 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:14.748010 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:14.748030 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:14.748067 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:14.749920 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:14.750793 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:14.750800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:14.750802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:14.750914 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:14.751273 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:14.751275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:14.751276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:14.751631 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:14.751777 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:14.752274 (+ 497us) server_negotiation.cc:299] Negotiation successful
0928 16:42:14.752480 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":136,"threads_started":1}
I20260928 16:42:14.752660 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.738330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:41307 (local address 127.0.0.1:39364)
0928 16:42:14.738869 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:14.738924 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:14.739214 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:14.740368 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:14.740375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:14.740913 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:14.741070 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:14.741080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.742537 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.742541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:14.743652 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:14.743679 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.744608 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.744612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:14.744766 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:14.745539 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:14.745559 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:14.747787 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:14.750100 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:14.750118 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:14.750125 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:14.750551 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:14.751037 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:14.751041 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:14.751043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:14.751158 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:14.751764 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:14.751771 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:14.751971 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 16:42:14.752154 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 16:42:14.752441 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":134,"threads_started":1}
I20260928 16:42:14.754202 7897 tablet_service.cc:1514] Processing DeleteTablet for tablet f2221ce185f64624b9a38e8f53356407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:42:14 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:39364
I20260928 16:42:14.755499 7972 tablet_replica.cc:330] stopping tablet replica
I20260928 16:42:14.755956 7972 raft_consensus.cc:2229] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:42:14.756338 7972 raft_consensus.cc:2258] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:42:14.757920 7972 ts_tablet_manager.cc:1905] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:42:14.763082 7972 ts_tablet_manager.cc:1918] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:42:14.763219 7972 log.cc:1196] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/wal/wals/f2221ce185f64624b9a38e8f53356407
I20260928 16:42:14.763511 7972 ts_tablet_manager.cc:1939] T f2221ce185f64624b9a38e8f53356407 P ca982d23e01e43f3ac61669de9c27699: Deleting consensus metadata
I20260928 16:42:14.766119 7761 catalog_manager.cc:4841] TS ca982d23e01e43f3ac61669de9c27699 (127.25.71.65:41307): tablet f2221ce185f64624b9a38e8f53356407 (table test-table [id=bfe5f5990a084fd0b08953dc2e0f9c0d]) successfully deleted
W20260928 16:42:14.846652 7965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:14.847000 7965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:14.847066 7965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:14.851411 7965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:14.851492 7965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:14.851579 7965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:42:14.857306 7965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:40907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=40665
--tserver_master_addrs=127.25.71.126:35439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:14.858590 7965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:14.859759 7965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:14.871136 7977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:14.871158 7980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:14.871220 7978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:14.871670 7965 server_base.cc:1034] running on GCE node
I20260928 16:42:14.872228 7965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:14.873104 7965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:14.874332 7965 hybrid_clock.cc:648] HybridClock initialized: now 1790613734874309 us; error 41 us; skew 500 ppm
Sep 28 16:42:14 dist-test-slave-nqgh krb5kdc[7193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613734, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:14.878060 7965 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:42:14.879410 7965 webserver.cc:460] Webserver started at http://127.25.71.66:40665/ using document root <none> and password file <none>
I20260928 16:42:14.880162 7965 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:14.880263 7965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:14.885685 7965 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:42:14.889197 7987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:14.891300 7965 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:42:14.891510 7965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "df42deab246142cbb05bff62c39fc80a"
format_stamp: "Formatted at 2026-09-28 16:42:13 on dist-test-slave-nqgh"
server_key: "fb2c593c842f352182f95136ff9a2104"
server_key_iv: "fb1596b8e2612b184fbf7243331869da"
server_key_version: "encryptionkey@0"
I20260928 16:42:14.892340 7965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:14.907281 7965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:14.911762 7965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:14.912009 7965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:14.912786 7965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:42:14.913915 7965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:42:14.913990 7965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:14.914065 7965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:42:14.914098 7965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:14.925472 7965 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:40907
I20260928 16:42:14.925542 8100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:40907 every 8 connection(s)
I20260928 16:42:14.926956 7965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:42:14.930105 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 7965
I20260928 16:42:14.930683 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:34275
--local_ip_for_outbound_sockets=127.25.71.67
--tserver_master_addrs=127.25.71.126:35439
--webserver_port=35167
--webserver_interface=127.25.71.67
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:42:14 dist-test-slave-nqgh krb5kdc[7193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613734, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:14.947708 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.930825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:35713 (local address 127.25.71.126:35439)
0928 16:42:14.931015 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:14.931020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:14.931658 (+ 638us) server_negotiation.cc:407] Connection header received
0928 16:42:14.933663 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:14.933669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:14.933768 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:14.933908 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:14.935620 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.936578 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.938031 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.938499 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.938681 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:14.942752 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:14.942757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:14.942760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:14.942803 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:14.944515 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:14.945503 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:14.945508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:14.945509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:14.945638 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:14.946135 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:14.946138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:14.946139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:14.946374 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:14.946460 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:14.947394 (+ 934us) server_negotiation.cc:299] Negotiation successful
0928 16:42:14.947561 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 16:42:14.949221 8103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:14.930210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.66:35713)
0928 16:42:14.931327 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:14.931421 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:14.933293 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:14.934147 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:14.934156 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:14.934698 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:14.935366 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:14.935388 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.936810 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.936813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:14.937851 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:14.937864 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:14.938754 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:14.938759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:14.938997 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:14.939814 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:14.939835 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:14.942316 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:14.944713 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:14.944725 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:14.944752 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:14.945309 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:14.945740 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:14.945743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:14.945745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:14.945976 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:14.946482 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:14.946491 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:14.947011 (+ 520us) client_negotiation.cc:769] Sending connection context
0928 16:42:14.947298 (+ 287us) client_negotiation.cc:240] Negotiation successful
0928 16:42:14.947795 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":207,"threads_started":1}
I20260928 16:42:14.951180 8101 heartbeater.cc:347] Connected to a master server at 127.25.71.126:35439
I20260928 16:42:14.951622 8101 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:14.952330 8101 heartbeater.cc:511] Master 127.25.71.126:35439 requested a full tablet report, sending...
I20260928 16:42:14.953807 7775 ts_manager.cc:194] Registered new tserver with Master: df42deab246142cbb05bff62c39fc80a (127.25.71.66:40907)
I20260928 16:42:14.954483 7775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:35713
W20260928 16:42:15.075400 8104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:15.075832 8104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:15.075904 8104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:15.080925 8104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:15.081045 8104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:15.081147 8104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:42:15.087940 8104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:34275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=35167
--tserver_master_addrs=127.25.71.126:35439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:15.089574 8104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:15.091037 8104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:15.104035 8109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:15.104048 8112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:15.104048 8110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:15.104574 8104 server_base.cc:1034] running on GCE node
I20260928 16:42:15.105129 8104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:15.105949 8104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:15.107286 8104 hybrid_clock.cc:648] HybridClock initialized: now 1790613735107256 us; error 50 us; skew 500 ppm
Sep 28 16:42:15 dist-test-slave-nqgh krb5kdc[7193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613735, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:15.110791 8104 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:42:15.111964 8104 webserver.cc:460] Webserver started at http://127.25.71.67:35167/ using document root <none> and password file <none>
I20260928 16:42:15.112689 8104 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:15.112788 8104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:15.117806 8104 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:42:15.120882 8119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:15.122838 8104 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:42:15.123020 8104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "0cbbadcc74d249c6aa3a29d6010cb055"
format_stamp: "Formatted at 2026-09-28 16:42:13 on dist-test-slave-nqgh"
server_key: "559cff2ce6b3b876694aca66e39e1f44"
server_key_iv: "15e1a9bf92617c5c37cc612ab5e017f4"
server_key_version: "encryptionkey@0"
I20260928 16:42:15.123690 8104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:15.161064 8104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:15.164980 8104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:15.165186 8104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:15.165838 8104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:42:15.166857 8104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:42:15.166923 8104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:15.166980 8104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:42:15.167008 8104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:15.177618 8104 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:34275
I20260928 16:42:15.177672 8232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:34275 every 8 connection(s)
I20260928 16:42:15.179100 8104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:42:15.181013 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 8104
I20260928 16:42:15.192159 8236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:15.181790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47330 (local address 127.25.71.126:35439)
0928 16:42:15.182114 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:15.182118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:15.182335 (+ 217us) server_negotiation.cc:407] Connection header received
0928 16:42:15.182501 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:15.182505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:15.182553 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:15.182654 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:15.183479 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.184040 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.185115 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.185526 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.185705 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:15.187655 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:15.187658 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:15.187661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:15.187696 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:15.189826 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:15.190475 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:15.190480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:15.190482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:15.190692 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:15.191074 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:15.191078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:15.191080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:15.191357 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:15.191618 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:15.191813 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 16:42:15.191977 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":203,"thread_start_us":111,"threads_started":1}
Sep 28 16:42:15 dist-test-slave-nqgh krb5kdc[7193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613735, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:15.198004 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:15.181790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:52053 (local address 127.25.71.126:35439)
0928 16:42:15.181946 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:15.181951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:15.183276 (+ 1325us) server_negotiation.cc:407] Connection header received
0928 16:42:15.184909 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:15.184912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:15.184971 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:15.185056 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:15.186932 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.187668 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.189657 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.189929 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.190068 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:15.193537 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:15.193541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:15.193544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:15.193570 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:15.195222 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:15.195984 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:15.195988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:15.195989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:15.196084 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:15.196552 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:15.196555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:15.196556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:15.196762 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:15.196863 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:15.197704 (+ 841us) server_negotiation.cc:299] Negotiation successful
0928 16:42:15.197852 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 16:42:15.199302 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:15.182427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.67:52053)
0928 16:42:15.183074 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:15.183140 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:15.184674 (+ 1534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:15.185325 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:15.185335 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:15.185945 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:15.186743 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:15.186766 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.187840 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.187844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:15.189457 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:15.189475 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.190260 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.190264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:15.190501 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:15.191244 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:15.191268 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:15.193345 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:15.195390 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:15.195398 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:15.195420 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:15.195818 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:15.196220 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:15.196223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:15.196224 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:15.196450 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:15.196863 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:15.196871 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:15.197401 (+ 530us) client_negotiation.cc:769] Sending connection context
0928 16:42:15.197616 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 16:42:15.198011 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":207,"threads_started":1}
I20260928 16:42:15.201428 8233 heartbeater.cc:347] Connected to a master server at 127.25.71.126:35439
I20260928 16:42:15.201792 8233 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:15.202687 8233 heartbeater.cc:511] Master 127.25.71.126:35439 requested a full tablet report, sending...
I20260928 16:42:15.204355 7775 ts_manager.cc:194] Registered new tserver with Master: 0cbbadcc74d249c6aa3a29d6010cb055 (127.25.71.67:34275)
I20260928 16:42:15.205219 7775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:52053
I20260928 16:42:15.206856 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:42:15.211649 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:15.207671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47344 (local address 127.25.71.126:35439)
0928 16:42:15.207827 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:15.207832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:15.208084 (+ 252us) server_negotiation.cc:407] Connection header received
0928 16:42:15.208262 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:15.208265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:15.208414 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:15.208508 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:42:15.208746 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.209256 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.210129 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.210339 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.210432 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:15.210794 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:42:15.211242 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:42:15.211353 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:15.211422 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 16:42:15.211488 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:42:15.212597 7770 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:42:15.217115 7770 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:42:15.228289 7897 tablet_service.cc:1467] Processing CreateTablet for tablet 7c2eff780a194fbf9b2ed88f4e5f79f5 (DEFAULT_TABLE table=test-table [id=81432f59f2a243aaa328910084a40607]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:42:15.229332 7897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c2eff780a194fbf9b2ed88f4e5f79f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:15.233407 7855 tablet_bootstrap.cc:492] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699: Bootstrap starting.
I20260928 16:42:15.234237 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:15.227002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:34275 (local address 127.0.0.1:37802)
0928 16:42:15.227158 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:15.227177 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:15.227305 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:15.229094 (+ 1789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:15.229097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:15.229118 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:15.229272 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:15.229280 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.230948 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.230951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:15.232227 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:15.232239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.233703 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.233708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:15.233857 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:15.233926 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 16:42:15.233983 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 16:42:15.234098 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 16:42:15.234946 7855 tablet_bootstrap.cc:654] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699: Neither blocks nor log segments found. Creating new log.
I20260928 16:42:15.235021 8242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:15.227288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37802 (local address 127.25.71.67:34275)
0928 16:42:15.228329 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:15.228345 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:15.228376 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:42:15.228483 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:15.228490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:15.228827 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:15.229099 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:15.229570 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.230789 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.232480 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.233518 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.234456 (+ 938us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:15.234652 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:15.234718 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 16:42:15.234861 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":922,"thread_start_us":223,"threads_started":1}
I20260928 16:42:15.235082 8243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:15.227412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:40907 (local address 127.0.0.1:49408)
0928 16:42:15.228008 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:15.228036 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:15.228409 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:15.229762 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:15.229765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:15.229785 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:15.229885 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:15.229891 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.231380 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.231384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:15.232475 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:15.232483 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.234686 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.234690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:15.234792 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:15.234816 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:42:15.234887 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 16:42:15.234964 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":119,"threads_started":1}
I20260928 16:42:15.235271 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:15.227987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49408 (local address 127.25.71.66:40907)
0928 16:42:15.228640 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:15.228655 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:15.228686 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:42:15.228786 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:15.228792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:15.229264 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:15.229611 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:15.230047 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.231205 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.232617 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.234482 (+ 1865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.234669 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:15.234864 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:15.234939 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 16:42:15.235075 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":161,"threads_started":1}
I20260928 16:42:15.239529 7855 tablet_bootstrap.cc:492] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699: No bootstrap required, opened a new log
I20260928 16:42:15.239688 7855 ts_tablet_manager.cc:1397] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 16:42:15.239147 8035 tablet_service.cc:1467] Processing CreateTablet for tablet 7c2eff780a194fbf9b2ed88f4e5f79f5 (DEFAULT_TABLE table=test-table [id=81432f59f2a243aaa328910084a40607]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:42:15.240448 7855 raft_consensus.cc:348] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:15.240635 7855 raft_consensus.cc:374] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:15.240696 7855 raft_consensus.cc:729] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca982d23e01e43f3ac61669de9c27699, State: Initialized, Role: FOLLOWER
I20260928 16:42:15.240823 8035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c2eff780a194fbf9b2ed88f4e5f79f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:15.240974 7855 consensus_queue.cc:260] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:15.241485 7855 ts_tablet_manager.cc:1428] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 16:42:15.241376 8167 tablet_service.cc:1467] Processing CreateTablet for tablet 7c2eff780a194fbf9b2ed88f4e5f79f5 (DEFAULT_TABLE table=test-table [id=81432f59f2a243aaa328910084a40607]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:42:15.242899 8167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c2eff780a194fbf9b2ed88f4e5f79f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:15.252657 8248 tablet_bootstrap.cc:492] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055: Bootstrap starting.
I20260928 16:42:15.252657 8247 tablet_bootstrap.cc:492] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a: Bootstrap starting.
I20260928 16:42:15.255548 8248 tablet_bootstrap.cc:654] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055: Neither blocks nor log segments found. Creating new log.
I20260928 16:42:15.256228 8247 tablet_bootstrap.cc:654] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a: Neither blocks nor log segments found. Creating new log.
I20260928 16:42:15.257413 8248 log.cc:824] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:15.257414 8247 log.cc:824] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:15.260720 8248 tablet_bootstrap.cc:492] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055: No bootstrap required, opened a new log
I20260928 16:42:15.260720 8247 tablet_bootstrap.cc:492] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a: No bootstrap required, opened a new log
I20260928 16:42:15.261053 8247 ts_tablet_manager.cc:1397] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 16:42:15.261053 8248 ts_tablet_manager.cc:1397] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 16:42:15.265990 8247 raft_consensus.cc:348] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:15.265990 8248 raft_consensus.cc:348] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:15.266427 8247 raft_consensus.cc:374] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:15.266427 8248 raft_consensus.cc:374] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:15.266494 8247 raft_consensus.cc:729] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df42deab246142cbb05bff62c39fc80a, State: Initialized, Role: FOLLOWER
I20260928 16:42:15.266494 8248 raft_consensus.cc:729] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cbbadcc74d249c6aa3a29d6010cb055, State: Initialized, Role: FOLLOWER
I20260928 16:42:15.267195 8248 consensus_queue.cc:260] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:15.267195 8247 consensus_queue.cc:260] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:15.268422 8233 heartbeater.cc:503] Master 127.25.71.126:35439 was elected leader, sending a full tablet report...
I20260928 16:42:15.268422 8101 heartbeater.cc:503] Master 127.25.71.126:35439 was elected leader, sending a full tablet report...
I20260928 16:42:15.268690 8248 ts_tablet_manager.cc:1428] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260928 16:42:15.268689 8247 ts_tablet_manager.cc:1428] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260928 16:42:15.418136 8246 raft_consensus.cc:482] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:42:15.418412 8246 raft_consensus.cc:504] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:15.419270 8251 raft_consensus.cc:482] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:42:15.419551 8251 raft_consensus.cc:504] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:15.419814 8246 leader_election.cc:288] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cbbadcc74d249c6aa3a29d6010cb055 (127.25.71.67:34275), df42deab246142cbb05bff62c39fc80a (127.25.71.66:40907)
I20260928 16:42:15.421281 8251 leader_election.cc:288] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df42deab246142cbb05bff62c39fc80a (127.25.71.66:40907), ca982d23e01e43f3ac61669de9c27699 (127.25.71.65:41307)
I20260928 16:42:15.426079 8242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:15.420500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:59645 (local address 127.25.71.67:34275)
0928 16:42:15.420633 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:15.420638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:15.420835 (+ 197us) server_negotiation.cc:407] Connection header received
0928 16:42:15.421483 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:15.421487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:15.421534 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:15.421607 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:15.422040 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.422837 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.424487 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.425163 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.425300 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:15.425370 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:15.425800 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 16:42:15.425896 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":53}
I20260928 16:42:15.426122 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:15.420373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:47855 (local address 127.25.71.66:40907)
0928 16:42:15.420530 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:15.420535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:15.420842 (+ 307us) server_negotiation.cc:407] Connection header received
0928 16:42:15.421085 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:15.421090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:15.421185 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:15.421315 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:15.422218 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.422981 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.424559 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.425163 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.425260 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:15.425321 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:15.425816 (+ 495us) server_negotiation.cc:299] Negotiation successful
0928 16:42:15.425965 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":55}
I20260928 16:42:15.426539 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:15.421559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:41307 (local address 127.25.71.67:58255)
0928 16:42:15.421811 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:15.421828 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:15.421901 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:15.422721 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:15.422725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:15.422757 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:15.422866 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:15.422872 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.423988 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.423992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:15.425245 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:15.425252 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.426147 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.426150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:15.426254 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:15.426276 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 16:42:15.426347 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 16:42:15.426424 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":1}
I20260928 16:42:15.426584 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:15.420046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:34275 (local address 127.25.71.65:59645)
0928 16:42:15.420696 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:15.420711 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:15.420878 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:15.421739 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:15.421743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:15.421772 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:15.421910 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:15.421919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.422978 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.422981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:15.424343 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:15.424355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.425433 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.425437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:15.425629 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:15.425677 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 16:42:15.426300 (+ 623us) client_negotiation.cc:240] Negotiation successful
0928 16:42:15.426409 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":64,"thread_start_us":108,"threads_started":1}
I20260928 16:42:15.427086 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:15.421711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:57373 (local address 127.25.71.66:40907)
0928 16:42:15.422211 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:15.422225 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:15.422546 (+ 321us) server_negotiation.cc:407] Connection header received
0928 16:42:15.422778 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:15.422781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:15.422833 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:15.422886 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:15.423328 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.424098 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.425748 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.426357 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.426445 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:15.426489 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:15.426888 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 16:42:15.426961 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":432,"thread_start_us":117,"threads_started":1}
I20260928 16:42:15.427138 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:15.420343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:40907 (local address 127.25.71.65:47855)
0928 16:42:15.420747 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:15.420759 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:15.420901 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:15.421861 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:15.421866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:15.421897 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:15.422046 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:15.422054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.423109 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.423112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:15.424423 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:15.424434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.425422 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.425425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:15.425568 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:15.425690 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 16:42:15.425864 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 16:42:15.426526 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":5,"thread_start_us":115,"threads_started":1}
I20260928 16:42:15.427556 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:15.421559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:40907 (local address 127.25.71.67:57373)
0928 16:42:15.422466 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:15.422481 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:15.422648 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:15.423017 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:15.423020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:15.423039 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:15.423178 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:15.423186 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.424240 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.424244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:15.425612 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:15.425623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.426602 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.426606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:15.426739 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:15.426770 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 16:42:15.427287 (+ 517us) client_negotiation.cc:240] Negotiation successful
0928 16:42:15.427399 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":498,"thread_start_us":165,"threads_started":1}
I20260928 16:42:15.427484 8187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2eff780a194fbf9b2ed88f4e5f79f5" candidate_uuid: "ca982d23e01e43f3ac61669de9c27699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" is_pre_election: true
I20260928 16:42:15.427800 8187 raft_consensus.cc:2454] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca982d23e01e43f3ac61669de9c27699 in term 0.
I20260928 16:42:15.428205 8055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2eff780a194fbf9b2ed88f4e5f79f5" candidate_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df42deab246142cbb05bff62c39fc80a" is_pre_election: true
W20260928 16:42:15.428404 8102 tablet.cc:2369] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:42:15.428599 8055 raft_consensus.cc:2454] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cbbadcc74d249c6aa3a29d6010cb055 in term 0.
I20260928 16:42:15.428797 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:15.421977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:58255 (local address 127.25.71.65:41307)
0928 16:42:15.422429 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:15.422445 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:15.422459 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:42:15.422499 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:15.422501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:15.422543 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:15.422955 (+ 412us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:15.423048 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.423845 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.425372 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.426014 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.428019 (+ 2005us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:15.428208 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:15.428252 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 16:42:15.428345 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":376,"thread_start_us":70,"threads_started":1}
I20260928 16:42:15.428992 7850 leader_election.cc:302] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cbbadcc74d249c6aa3a29d6010cb055, ca982d23e01e43f3ac61669de9c27699; no voters:
I20260928 16:42:15.429206 8120 leader_election.cc:302] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cbbadcc74d249c6aa3a29d6010cb055, df42deab246142cbb05bff62c39fc80a; no voters:
I20260928 16:42:15.429370 8246 raft_consensus.cc:2781] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:42:15.429484 8246 raft_consensus.cc:482] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:42:15.429548 8246 raft_consensus.cc:3037] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:15.429504 8251 raft_consensus.cc:2781] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:42:15.429589 8251 raft_consensus.cc:482] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:42:15.429630 8251 raft_consensus.cc:3037] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:15.429718 7916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2eff780a194fbf9b2ed88f4e5f79f5" candidate_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca982d23e01e43f3ac61669de9c27699" is_pre_election: true
I20260928 16:42:15.430038 8055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2eff780a194fbf9b2ed88f4e5f79f5" candidate_uuid: "ca982d23e01e43f3ac61669de9c27699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df42deab246142cbb05bff62c39fc80a" is_pre_election: true
I20260928 16:42:15.430183 8055 raft_consensus.cc:2454] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca982d23e01e43f3ac61669de9c27699 in term 0.
W20260928 16:42:15.430480 8234 tablet.cc:2369] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:42:15.431237 8246 raft_consensus.cc:504] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:15.431237 8251 raft_consensus.cc:504] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:15.431612 7916 raft_consensus.cc:2379] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cbbadcc74d249c6aa3a29d6010cb055 in current term 1: Already voted for candidate ca982d23e01e43f3ac61669de9c27699 in this term.
I20260928 16:42:15.431754 8251 leader_election.cc:288] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [CANDIDATE]: Term 1 election: Requested vote from peers df42deab246142cbb05bff62c39fc80a (127.25.71.66:40907), ca982d23e01e43f3ac61669de9c27699 (127.25.71.65:41307)
I20260928 16:42:15.431752 8246 leader_election.cc:288] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 1 election: Requested vote from peers 0cbbadcc74d249c6aa3a29d6010cb055 (127.25.71.67:34275), df42deab246142cbb05bff62c39fc80a (127.25.71.66:40907)
I20260928 16:42:15.432363 8055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2eff780a194fbf9b2ed88f4e5f79f5" candidate_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df42deab246142cbb05bff62c39fc80a"
I20260928 16:42:15.432416 8187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2eff780a194fbf9b2ed88f4e5f79f5" candidate_uuid: "ca982d23e01e43f3ac61669de9c27699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cbbadcc74d249c6aa3a29d6010cb055"
I20260928 16:42:15.432601 8187 raft_consensus.cc:2379] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ca982d23e01e43f3ac61669de9c27699 in current term 1: Already voted for candidate 0cbbadcc74d249c6aa3a29d6010cb055 in this term.
I20260928 16:42:15.432533 8055 raft_consensus.cc:3037] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:15.432829 8054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2eff780a194fbf9b2ed88f4e5f79f5" candidate_uuid: "ca982d23e01e43f3ac61669de9c27699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df42deab246142cbb05bff62c39fc80a"
I20260928 16:42:15.433461 7916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2eff780a194fbf9b2ed88f4e5f79f5" candidate_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca982d23e01e43f3ac61669de9c27699"
I20260928 16:42:15.433508 7849 leader_election.cc:302] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca982d23e01e43f3ac61669de9c27699; no voters: 0cbbadcc74d249c6aa3a29d6010cb055, df42deab246142cbb05bff62c39fc80a
I20260928 16:42:15.433650 7916 raft_consensus.cc:2379] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0cbbadcc74d249c6aa3a29d6010cb055 in current term 1: Already voted for candidate ca982d23e01e43f3ac61669de9c27699 in this term.
I20260928 16:42:15.433806 8246 raft_consensus.cc:2726] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 16:42:15.434216 8055 raft_consensus.cc:2454] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cbbadcc74d249c6aa3a29d6010cb055 in term 1.
I20260928 16:42:15.434587 8120 leader_election.cc:302] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cbbadcc74d249c6aa3a29d6010cb055, df42deab246142cbb05bff62c39fc80a; no voters: ca982d23e01e43f3ac61669de9c27699
I20260928 16:42:15.434875 8251 raft_consensus.cc:2781] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:42:15.435173 8251 raft_consensus.cc:686] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 1 LEADER]: Becoming Leader. State: Replica: 0cbbadcc74d249c6aa3a29d6010cb055, State: Running, Role: LEADER
I20260928 16:42:15.435629 8251 consensus_queue.cc:237] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:15.439785 7775 catalog_manager.cc:5495] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cbbadcc74d249c6aa3a29d6010cb055 (127.25.71.67). New cstate: current_term: 1 leader_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } health_report { overall_health: UNKNOWN } } }
W20260928 16:42:15.459321 7964 tablet.cc:2369] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:42:15.487200 8242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:15.483116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.25.71.67:34275)
0928 16:42:15.483334 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:15.483339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:15.483358 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:42:15.483478 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:15.483481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:15.483553 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:15.483665 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:42:15.483950 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.484499 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.485395 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:15.485619 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:15.485712 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:15.486079 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:42:15.486741 (+ 662us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:42:15.486863 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:15.486925 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 16:42:15.487001 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260928 16:42:15.494997 7916 raft_consensus.cc:1261] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Refusing update from remote peer 0cbbadcc74d249c6aa3a29d6010cb055: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:42:15.495078 8055 raft_consensus.cc:1261] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a [term 1 FOLLOWER]: Refusing update from remote peer 0cbbadcc74d249c6aa3a29d6010cb055: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:42:15.496021 8258 consensus_queue.cc:1035] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:42:15.496402 8251 consensus_queue.cc:1035] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:42:15.507671 8263 mvcc.cc:205] Tried to move back new op lower bound from 7334353860577931264 to 7334353860347813888. Current Snapshot: MvccSnapshot[applied={T|T < 7334353860577931264}]
I20260928 16:42:15.518527 7770 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:42:15.518834 7770 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:42:15.523057 7770 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647: Sending DeleteTablet for 3 replicas of tablet 7c2eff780a194fbf9b2ed88f4e5f79f5
I20260928 16:42:15.524035 7897 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c2eff780a194fbf9b2ed88f4e5f79f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:42:15 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:39364
I20260928 16:42:15.524132 8167 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c2eff780a194fbf9b2ed88f4e5f79f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:42:15 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:37802
I20260928 16:42:15.524227 8035 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c2eff780a194fbf9b2ed88f4e5f79f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:42:15 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:49408
I20260928 16:42:15.525029 8271 tablet_replica.cc:330] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055: stopping tablet replica
I20260928 16:42:15.525144 8270 tablet_replica.cc:330] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699: stopping tablet replica
I20260928 16:42:15.525168 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 7813
I20260928 16:42:15.525300 8270 raft_consensus.cc:2229] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:42:15.525692 8271 raft_consensus.cc:2229] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:42:15.526293 8271 raft_consensus.cc:2258] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:42:15.526875 8272 tablet_replica.cc:330] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a: stopping tablet replica
I20260928 16:42:15.527364 8272 raft_consensus.cc:2229] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:42:15.527813 8272 raft_consensus.cc:2258] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:42:15.529814 8272 ts_tablet_manager.cc:1905] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:42:15.529814 8271 ts_tablet_manager.cc:1905] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:42:15.533401 8272 ts_tablet_manager.cc:1918] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:42:15.533401 8271 ts_tablet_manager.cc:1918] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:42:15.533569 8272 log.cc:1196] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/7c2eff780a194fbf9b2ed88f4e5f79f5
I20260928 16:42:15.533569 8271 log.cc:1196] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/7c2eff780a194fbf9b2ed88f4e5f79f5
I20260928 16:42:15.533948 8272 ts_tablet_manager.cc:1939] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P df42deab246142cbb05bff62c39fc80a: Deleting consensus metadata
I20260928 16:42:15.533948 8271 ts_tablet_manager.cc:1939] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P 0cbbadcc74d249c6aa3a29d6010cb055: Deleting consensus metadata
I20260928 16:42:15.536078 7759 catalog_manager.cc:4841] TS df42deab246142cbb05bff62c39fc80a (127.25.71.66:40907): tablet 7c2eff780a194fbf9b2ed88f4e5f79f5 (table test-table [id=81432f59f2a243aaa328910084a40607]) successfully deleted
I20260928 16:42:15.536078 7760 catalog_manager.cc:4841] TS 0cbbadcc74d249c6aa3a29d6010cb055 (127.25.71.67:34275): tablet 7c2eff780a194fbf9b2ed88f4e5f79f5 (table test-table [id=81432f59f2a243aaa328910084a40607]) successfully deleted
W20260928 16:42:15.540313 7761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.71.65:41307 (error 108)
I20260928 16:42:15.540632 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 7965
I20260928 16:42:15.542377 8243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:15.542022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:41307 (local address 127.0.0.1:39372)
0928 16:42:15.542193 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:15.542272 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.65:41307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 16:42:15.542610 7761 catalog_manager.cc:4584] TS ca982d23e01e43f3ac61669de9c27699 (127.25.71.65:41307): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7c2eff780a194fbf9b2ed88f4e5f79f5: Network error: Client connection negotiation failed: client connection to 127.25.71.65:41307: connect: Connection refused (error 111)
I20260928 16:42:15.549266 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 8104
I20260928 16:42:15.555575 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 7742
I20260928 16:42:15.564320 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:35439
--webserver_interface=127.25.71.126
--webserver_port=43541
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:35439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:42:15.706247 8274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:15.706708 8274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:15.706779 8274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:15.711594 8274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:42:15.711740 8274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:15.711774 8274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:15.711797 8274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:42:15.711819 8274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:42:15.719743 8274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:35439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:35439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=43541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:15.721455 8274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:15.722986 8274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:15.734102 8280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:15.734200 8279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:15.734169 8274 server_base.cc:1034] running on GCE node
W20260928 16:42:15.734511 8282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:15.735350 8274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:15.736918 8274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:15.738121 8274 hybrid_clock.cc:648] HybridClock initialized: now 1790613735738070 us; error 73 us; skew 500 ppm
Sep 28 16:42:15 dist-test-slave-nqgh krb5kdc[7193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613735, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:15.742261 8274 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:42:15.743904 8274 webserver.cc:460] Webserver started at http://127.25.71.126:43541/ using document root <none> and password file <none>
I20260928 16:42:15.744719 8274 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:15.744815 8274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:15.750396 8274 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:42:15.753806 8289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:15.755769 8274 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:42:15.755970 8274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "86249db880874e95aac51f00486b0647"
format_stamp: "Formatted at 2026-09-28 16:42:13 on dist-test-slave-nqgh"
server_key: "29ffac6c5f67129f2d3efec71e3eb3d0"
server_key_iv: "cd085ade24946f29250c7465da44293d"
server_key_version: "encryptionkey@0"
I20260928 16:42:15.756829 8274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:15.778966 8274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:15.784359 8274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:15.784669 8274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:15.795744 8274 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:35439
I20260928 16:42:15.796104 8341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:35439 every 8 connection(s)
I20260928 16:42:15.797608 8274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:42:15.802964 8342 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:42:15.804456 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 8274
I20260928 16:42:15.805087 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:41307
--local_ip_for_outbound_sockets=127.25.71.65
--tserver_master_addrs=127.25.71.126:35439
--webserver_port=34423
--webserver_interface=127.25.71.65
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:42:15.805290 8342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647: Bootstrap starting.
I20260928 16:42:15.824541 8342 log.cc:824] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:15.837244 8342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:42:15.837704 8342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647: Bootstrap complete.
I20260928 16:42:15.842161 8342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86249db880874e95aac51f00486b0647" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35439 } }
I20260928 16:42:15.842665 8342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86249db880874e95aac51f00486b0647, State: Initialized, Role: FOLLOWER
I20260928 16:42:15.843246 8342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86249db880874e95aac51f00486b0647" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35439 } }
I20260928 16:42:15.843377 8342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:42:15.843457 8342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:42:15.843567 8342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 2 FOLLOWER]: Advancing to term 3
I20260928 16:42:15.845855 8342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86249db880874e95aac51f00486b0647" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35439 } }
I20260928 16:42:15.846239 8342 leader_election.cc:302] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86249db880874e95aac51f00486b0647; no voters:
I20260928 16:42:15.846606 8342 leader_election.cc:288] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 16:42:15.846753 8346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 16:42:15.846987 8346 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [term 3 LEADER]: Becoming Leader. State: Replica: 86249db880874e95aac51f00486b0647, State: Running, Role: LEADER
I20260928 16:42:15.847353 8346 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86249db880874e95aac51f00486b0647" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35439 } }
I20260928 16:42:15.848173 8342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:42:15.848834 8347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "86249db880874e95aac51f00486b0647" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86249db880874e95aac51f00486b0647" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35439 } } }
I20260928 16:42:15.848985 8347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [sys.catalog]: This master's current role is: LEADER
I20260928 16:42:15.849249 8347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86249db880874e95aac51f00486b0647. Latest consensus state: current_term: 3 leader_uuid: "86249db880874e95aac51f00486b0647" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86249db880874e95aac51f00486b0647" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35439 } } }
I20260928 16:42:15.849363 8347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86249db880874e95aac51f00486b0647 [sys.catalog]: This master's current role is: LEADER
I20260928 16:42:15.850426 8352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:42:15.855604 8352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:42:15.856246 8352 catalog_manager.cc:1259] Loaded cluster ID: 7ddadfa61a1d4f54a7cea9833dacd8bb
I20260928 16:42:15.856308 8352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 16:42:15.857262 8352 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f83c2a099ad google::LogMessage::Fail() at ??:0
@ 0x7f83c2a0c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f83c2a094de google::LogMessage::Flush() at ??:0
@ 0x7f83c2a0aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f83cd34cfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f83cd34dd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f83cd34a579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f83cd38e640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f83c4c5b3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f83c4c5bc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f83c4c5d3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f83c4c4c512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f83c62906db start_thread at ??:0
@ 0x7f83c150761f clone at ??:0
W20260928 16:42:15.987519 8344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:15.987908 8344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:15.987960 8344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:15.992883 8344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:15.993003 8344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:15.993077 8344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:42:15.999657 8344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:41307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=34423
--tserver_master_addrs=127.25.71.126:35439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:16.001214 8344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:16.002538 8344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:16.015148 8384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:16.015162 8387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:16.015630 8344 server_base.cc:1034] running on GCE node
W20260928 16:42:16.015890 8385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:16.016444 8344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:16.017341 8344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:16.018538 8344 hybrid_clock.cc:648] HybridClock initialized: now 1790613736018498 us; error 65 us; skew 500 ppm
Sep 28 16:42:16 dist-test-slave-nqgh krb5kdc[7193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613736, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:16.022359 8344 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:42:16.023690 8344 webserver.cc:460] Webserver started at http://127.25.71.65:34423/ using document root <none> and password file <none>
I20260928 16:42:16.024443 8344 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:16.024529 8344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:16.030193 8344 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:42:16.033810 8394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:16.036221 8344 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 16:42:16.036499 8344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "ca982d23e01e43f3ac61669de9c27699"
format_stamp: "Formatted at 2026-09-28 16:42:13 on dist-test-slave-nqgh"
server_key: "475a775c9738192e1ebba366a102a028"
server_key_iv: "ed02149879c73cfef3a6743be27feedc"
server_key_version: "encryptionkey@0"
I20260928 16:42:16.037304 8344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:16.072506 8344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:16.077272 8344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:16.077517 8344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:16.078325 8344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:42:16.079867 8401 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 16:42:16.085482 8344 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 16:42:16.085587 8344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260928 16:42:16.085656 8344 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 16:42:16.088029 8344 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 16:42:16.088121 8344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260928 16:42:16.088646 8401 tablet_bootstrap.cc:492] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699: Bootstrap starting.
I20260928 16:42:16.103417 8344 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:41307
I20260928 16:42:16.103883 8508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:41307 every 8 connection(s)
I20260928 16:42:16.104909 8344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:42:16.105911 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 8344
I20260928 16:42:16.106422 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:40907
--local_ip_for_outbound_sockets=127.25.71.66
--tserver_master_addrs=127.25.71.126:35439
--webserver_port=40665
--webserver_interface=127.25.71.66
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:42:16.111815 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:16.108371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:47601)
0928 16:42:16.110147 (+ 1776us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:16.110355 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1447,"thread_start_us":253,"threads_started":1}
W20260928 16:42:16.112777 8509 heartbeater.cc:644] Failed to heartbeat to 127.25.71.126:35439 (0 consecutive failures): Network error: Failed to ping master at 127.25.71.126:35439: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
I20260928 16:42:16.113901 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:16.113470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:50073)
0928 16:42:16.113681 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:16.113787 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 16:42:16.115089 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:16.114802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:40583)
0928 16:42:16.114944 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:16.115005 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 16:42:16.115432 8509 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 16:42:16.115746 8401 log.cc:824] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:16.124063 8401 tablet_bootstrap.cc:492] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:42:16.124454 8401 tablet_bootstrap.cc:492] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699: Bootstrap complete.
I20260928 16:42:16.125509 8401 ts_tablet_manager.cc:1397] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699: Time spent bootstrapping tablet: real 0.037s user 0.016s sys 0.017s
I20260928 16:42:16.128474 8401 raft_consensus.cc:348] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:16.128974 8401 raft_consensus.cc:729] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca982d23e01e43f3ac61669de9c27699, State: Initialized, Role: FOLLOWER
I20260928 16:42:16.129470 8401 consensus_queue.cc:260] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:16.130530 8401 ts_tablet_manager.cc:1428] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260928 16:42:16.131448 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:16.130983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:36773)
0928 16:42:16.131189 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:16.131297 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 16:42:16.132465 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:16.132113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:35131)
0928 16:42:16.132326 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:16.132393 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260928 16:42:16.259994 8511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:16.260412 8511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:16.260473 8511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:16.265654 8511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:16.265782 8511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:16.265856 8511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:42:16.272725 8511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:40907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=40665
--tserver_master_addrs=127.25.71.126:35439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:16.274324 8511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:16.275775 8511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:42:16.290431 8511 server_base.cc:1034] running on GCE node
W20260928 16:42:16.290318 8519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:16.290318 8520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:16.290313 8522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:16.291340 8511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:16.292249 8511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:16.293453 8511 hybrid_clock.cc:648] HybridClock initialized: now 1790613736293422 us; error 50 us; skew 500 ppm
Sep 28 16:42:16 dist-test-slave-nqgh krb5kdc[7193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613736, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:16.297287 8511 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:42:16.298789 8511 webserver.cc:460] Webserver started at http://127.25.71.66:40665/ using document root <none> and password file <none>
I20260928 16:42:16.299669 8511 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:16.299772 8511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:16.305526 8511 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:42:16.308693 8529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:16.310477 8511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:42:16.310737 8511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "df42deab246142cbb05bff62c39fc80a"
format_stamp: "Formatted at 2026-09-28 16:42:13 on dist-test-slave-nqgh"
server_key: "fb2c593c842f352182f95136ff9a2104"
server_key_iv: "fb1596b8e2612b184fbf7243331869da"
server_key_version: "encryptionkey@0"
I20260928 16:42:16.311547 8511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:16.328215 8511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:16.332921 8511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:16.333189 8511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:16.334035 8511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:42:16.335238 8511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:42:16.335297 8511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:16.335363 8511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:42:16.335391 8511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:16.347816 8511 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:40907
I20260928 16:42:16.347847 8642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:40907 every 8 connection(s)
I20260928 16:42:16.349561 8511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:42:16.355206 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:16.352857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.66:56589)
0928 16:42:16.353641 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:16.353835 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":177,"threads_started":1}
W20260928 16:42:16.355866 8643 heartbeater.cc:644] Failed to heartbeat to 127.25.71.126:35439 (0 consecutive failures): Network error: Failed to ping master at 127.25.71.126:35439: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
I20260928 16:42:16.356890 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:16.356527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.66:60649)
0928 16:42:16.356729 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:16.356813 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 16:42:16.357851 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:16.357553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.66:53513)
0928 16:42:16.357725 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:16.357785 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260928 16:42:16.358120 8643 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 16:42:16.358731 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 8511
I20260928 16:42:16.359310 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:34275
--local_ip_for_outbound_sockets=127.25.71.67
--tserver_master_addrs=127.25.71.126:35439
--webserver_port=35167
--webserver_interface=127.25.71.67
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:42:16.500823 8646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:16.501230 8646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:16.501283 8646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:16.506448 8646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:16.506585 8646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:16.506706 8646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:42:16.513655 8646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:34275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=35167
--tserver_master_addrs=127.25.71.126:35439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:16.515197 8646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:16.516748 8646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:16.531327 8654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:16.531277 8651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:16.531358 8652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:16.531528 8646 server_base.cc:1034] running on GCE node
I20260928 16:42:16.532286 8646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:16.533210 8646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:16.534433 8646 hybrid_clock.cc:648] HybridClock initialized: now 1790613736534412 us; error 45 us; skew 500 ppm
Sep 28 16:42:16 dist-test-slave-nqgh krb5kdc[7193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613736, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:16.538213 8646 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:42:16.539713 8646 webserver.cc:460] Webserver started at http://127.25.71.67:35167/ using document root <none> and password file <none>
I20260928 16:42:16.540449 8646 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:16.540527 8646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:16.546124 8646 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:42:16.549315 8661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:16.551697 8646 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:42:16.552294 8646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "0cbbadcc74d249c6aa3a29d6010cb055"
format_stamp: "Formatted at 2026-09-28 16:42:13 on dist-test-slave-nqgh"
server_key: "559cff2ce6b3b876694aca66e39e1f44"
server_key_iv: "15e1a9bf92617c5c37cc612ab5e017f4"
server_key_version: "encryptionkey@0"
I20260928 16:42:16.553218 8646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:16.571168 8646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:16.575980 8646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:16.576222 8646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:16.577057 8646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:42:16.578231 8646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:42:16.578293 8646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:16.578348 8646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:42:16.578372 8646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:16.590075 8646 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:34275
I20260928 16:42:16.590124 8774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:34275 every 8 connection(s)
I20260928 16:42:16.591511 8646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:42:16.596978 8777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:16.594483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.67:36425)
0928 16:42:16.595314 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:16.595514 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":179,"threads_started":1}
W20260928 16:42:16.597704 8775 heartbeater.cc:644] Failed to heartbeat to 127.25.71.126:35439 (0 consecutive failures): Network error: Failed to ping master at 127.25.71.126:35439: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
I20260928 16:42:16.598825 8777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:16.598401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.67:41185)
0928 16:42:16.598621 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:16.598751 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260928 16:42:16.599412 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 8646
I20260928 16:42:16.599776 8777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:16.599489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.67:43913)
0928 16:42:16.599633 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:16.599702 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 16:42:16.599989 8775 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 16:42:17.134388 8779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:17.133656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:59197)
0928 16:42:17.134170 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:17.134260 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":117,"threads_started":1}
I20260928 16:42:17.359786 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:17.359086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.66:49203)
0928 16:42:17.359487 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:17.359578 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":108,"threads_started":1}
I20260928 16:42:17.418960 8781 raft_consensus.cc:482] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:42:17.419298 8781 raft_consensus.cc:504] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:17.420881 8781 leader_election.cc:288] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cbbadcc74d249c6aa3a29d6010cb055 (127.25.71.67:34275), df42deab246142cbb05bff62c39fc80a (127.25.71.66:40907)
Sep 28 16:42:17 dist-test-slave-nqgh krb5kdc[7193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613736, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.67@KRBTEST.COM
Sep 28 16:42:17 dist-test-slave-nqgh krb5kdc[7193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613736, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.66@KRBTEST.COM
I20260928 16:42:17.444775 8783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:17.421515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:40545 (local address 127.25.71.66:40907)
0928 16:42:17.422197 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:17.422224 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:17.422278 (+ 54us) server_negotiation.cc:407] Connection header received
0928 16:42:17.423580 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:17.423590 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:17.424047 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:17.424427 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:17.425730 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:17.426959 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:17.428532 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:17.429083 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:17.429267 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:17.437917 (+ 8650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:17.437924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:17.437946 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:17.437992 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:17.440397 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:17.442061 (+ 1664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:17.442067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:17.442074 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:17.442232 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:17.442683 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:17.442687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:17.442689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:17.443327 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:17.443466 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:17.444111 (+ 645us) server_negotiation.cc:299] Negotiation successful
0928 16:42:17.444496 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":123,"threads_started":1}
I20260928 16:42:17.444758 8779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:17.421254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:40907 (local address 127.25.71.65:40545)
0928 16:42:17.421511 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:17.421549 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:17.422258 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:17.424408 (+ 2150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:17.424412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:17.424764 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:17.425495 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:17.425515 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:17.427136 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:17.427140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:17.428368 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:17.428382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:17.429225 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:17.429229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:17.429489 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:17.434660 (+ 5171us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:17.434699 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:17.437541 (+ 2842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:17.440665 (+ 3124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:17.440676 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:17.440700 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:17.441317 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:17.442383 (+ 1066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:17.442387 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:17.442389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:17.442522 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:17.443461 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:17.443464 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:17.443736 (+ 272us) client_negotiation.cc:769] Sending connection context
0928 16:42:17.443953 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 16:42:17.444512 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":394,"spinlock_wait_cycles":214912}
I20260928 16:42:17.445060 8782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:17.421289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:34275 (local address 127.25.71.65:44683)
0928 16:42:17.421798 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:17.421814 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:17.422280 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:17.424117 (+ 1837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:17.424126 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:17.424766 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:17.425492 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:17.425515 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:17.427066 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:17.427069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:17.428372 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:17.428387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:17.429210 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:17.429214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:17.429486 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:17.434660 (+ 5174us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:17.434699 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:17.437244 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:17.440665 (+ 3421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:17.440676 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:17.440700 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:17.441317 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:17.441989 (+ 672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:17.441993 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:17.441995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:17.442324 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:17.443411 (+ 1087us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:17.443419 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:17.443737 (+ 318us) client_negotiation.cc:769] Sending connection context
0928 16:42:17.443954 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 16:42:17.444635 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":305,"spinlock_wait_cycles":51456,"thread_start_us":100,"threads_started":1}
I20260928 16:42:17.445307 8784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:17.421554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:44683 (local address 127.25.71.67:34275)
0928 16:42:17.422111 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:17.422129 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:17.422184 (+ 55us) server_negotiation.cc:407] Connection header received
0928 16:42:17.423277 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:17.423284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:17.423694 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:17.423997 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:17.425726 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:17.426900 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:17.428520 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:17.429053 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:17.429225 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:17.437636 (+ 8411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:17.437643 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:17.437663 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:17.437726 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:17.440397 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:17.441527 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:17.441536 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:17.441546 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:17.441804 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:17.442458 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:17.442463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:17.442465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:17.443254 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:17.443411 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:17.444124 (+ 713us) server_negotiation.cc:299] Negotiation successful
0928 16:42:17.444595 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":210,"threads_started":1}
W20260928 16:42:17.448565 8395 leader_election.cc:341] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df42deab246142cbb05bff62c39fc80a (127.25.71.66:40907): Not found: Tablet not found: 7c2eff780a194fbf9b2ed88f4e5f79f5
W20260928 16:42:17.448796 8396 leader_election.cc:341] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cbbadcc74d249c6aa3a29d6010cb055 (127.25.71.67:34275): Not found: Tablet not found: 7c2eff780a194fbf9b2ed88f4e5f79f5
I20260928 16:42:17.448879 8396 leader_election.cc:302] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca982d23e01e43f3ac61669de9c27699; no voters: 0cbbadcc74d249c6aa3a29d6010cb055, df42deab246142cbb05bff62c39fc80a
I20260928 16:42:17.449268 8781 raft_consensus.cc:2726] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:42:17.601653 8785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:17.600733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.67:51091)
0928 16:42:17.601321 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:17.601439 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":99,"threads_started":1}
I20260928 16:42:18.136441 8786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:18.135778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:38161)
0928 16:42:18.136229 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:18.136322 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":107,"threads_started":1}
I20260928 16:42:18.361585 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:18.360807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.66:42331)
0928 16:42:18.361334 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:18.361438 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":91,"threads_started":1}
I20260928 16:42:18.603601 8788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:18.602935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.67:42967)
0928 16:42:18.603344 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:18.603437 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":103,"threads_started":1}
I20260928 16:42:18.987584 8789 raft_consensus.cc:482] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:42:18.987761 8789 raft_consensus.cc:504] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:18.988304 8789 leader_election.cc:288] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cbbadcc74d249c6aa3a29d6010cb055 (127.25.71.67:34275), df42deab246142cbb05bff62c39fc80a (127.25.71.66:40907)
W20260928 16:42:18.989542 8396 leader_election.cc:341] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cbbadcc74d249c6aa3a29d6010cb055 (127.25.71.67:34275): Not found: Tablet not found: 7c2eff780a194fbf9b2ed88f4e5f79f5
W20260928 16:42:18.989701 8395 leader_election.cc:341] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df42deab246142cbb05bff62c39fc80a (127.25.71.66:40907): Not found: Tablet not found: 7c2eff780a194fbf9b2ed88f4e5f79f5
I20260928 16:42:18.989814 8395 leader_election.cc:302] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca982d23e01e43f3ac61669de9c27699; no voters: 0cbbadcc74d249c6aa3a29d6010cb055, df42deab246142cbb05bff62c39fc80a
I20260928 16:42:18.990052 8789 raft_consensus.cc:2726] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:42:19.138358 8790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:19.137633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:38919)
0928 16:42:19.138156 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:19.138245 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":105,"threads_started":1}
I20260928 16:42:19.363288 8791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:19.362763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.66:37491)
0928 16:42:19.363093 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:19.363161 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":154,"threads_started":1}
I20260928 16:42:19.605345 8792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:19.604746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.67:55295)
0928 16:42:19.605072 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:19.605158 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":79,"threads_started":1}
I20260928 16:42:20.140497 8793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:20.139863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:58697)
0928 16:42:20.140289 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:20.140356 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":125,"threads_started":1}
I20260928 16:42:20.364877 8794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:20.364207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.66:37365)
0928 16:42:20.364639 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:20.364740 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":95,"threads_started":1}
I20260928 16:42:20.607247 8795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:20.606527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.67:42069)
0928 16:42:20.606955 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:20.607052 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":139,"threads_started":1}
I20260928 16:42:21.142379 8796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:21.141770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:56085)
0928 16:42:21.142152 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:21.142235 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":101,"threads_started":1}
I20260928 16:42:21.174089 8797 raft_consensus.cc:482] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:42:21.174256 8797 raft_consensus.cc:504] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:21.174708 8797 leader_election.cc:288] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cbbadcc74d249c6aa3a29d6010cb055 (127.25.71.67:34275), df42deab246142cbb05bff62c39fc80a (127.25.71.66:40907)
W20260928 16:42:21.175740 8396 leader_election.cc:341] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cbbadcc74d249c6aa3a29d6010cb055 (127.25.71.67:34275): Not found: Tablet not found: 7c2eff780a194fbf9b2ed88f4e5f79f5
W20260928 16:42:21.175843 8395 leader_election.cc:341] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df42deab246142cbb05bff62c39fc80a (127.25.71.66:40907): Not found: Tablet not found: 7c2eff780a194fbf9b2ed88f4e5f79f5
I20260928 16:42:21.175911 8395 leader_election.cc:302] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca982d23e01e43f3ac61669de9c27699; no voters: 0cbbadcc74d249c6aa3a29d6010cb055, df42deab246142cbb05bff62c39fc80a
I20260928 16:42:21.176066 8797 raft_consensus.cc:2726] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:42:21.367151 8798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:21.366279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.66:49391)
0928 16:42:21.366745 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:21.366860 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":104,"threads_started":1}
I20260928 16:42:21.609050 8799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:21.608411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.67:35865)
0928 16:42:21.608805 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:21.608874 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":104,"threads_started":1}
I20260928 16:42:22.144301 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:22.143533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:41379)
0928 16:42:22.143951 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:22.144145 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":164,"threads_started":1}
I20260928 16:42:22.369135 8801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:22.368452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.66:33591)
0928 16:42:22.368890 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:22.368988 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":94,"threads_started":1}
I20260928 16:42:22.610983 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:22.610241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.67:60651)
0928 16:42:22.610706 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:22.610795 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":115,"threads_started":1}
I20260928 16:42:23.146262 8803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:23.145569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:52951)
0928 16:42:23.145956 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:23.146053 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":90,"threads_started":1}
I20260928 16:42:23.371078 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:23.370323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.66:43077)
0928 16:42:23.370830 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:23.370931 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":93,"threads_started":1}
I20260928 16:42:23.612780 8805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:23.612054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.67:41301)
0928 16:42:23.612535 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:23.612631 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":98,"threads_started":1}
I20260928 16:42:23.993089 8806 raft_consensus.cc:482] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:42:23.993263 8806 raft_consensus.cc:504] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:23.993822 8806 leader_election.cc:288] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cbbadcc74d249c6aa3a29d6010cb055 (127.25.71.67:34275), df42deab246142cbb05bff62c39fc80a (127.25.71.66:40907)
W20260928 16:42:23.995007 8396 leader_election.cc:341] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cbbadcc74d249c6aa3a29d6010cb055 (127.25.71.67:34275): Not found: Tablet not found: 7c2eff780a194fbf9b2ed88f4e5f79f5
W20260928 16:42:23.995214 8395 leader_election.cc:341] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df42deab246142cbb05bff62c39fc80a (127.25.71.66:40907): Not found: Tablet not found: 7c2eff780a194fbf9b2ed88f4e5f79f5
I20260928 16:42:23.995337 8395 leader_election.cc:302] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca982d23e01e43f3ac61669de9c27699; no voters: 0cbbadcc74d249c6aa3a29d6010cb055, df42deab246142cbb05bff62c39fc80a
I20260928 16:42:23.995544 8806 raft_consensus.cc:2726] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:42:24.148149 8807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:24.147475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:41473)
0928 16:42:24.147843 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:24.147926 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":93,"threads_started":1}
I20260928 16:42:24.373143 8808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:24.372430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.66:42675)
0928 16:42:24.372927 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:24.373024 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":112,"threads_started":1}
I20260928 16:42:24.614694 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:24.613967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.67:35927)
0928 16:42:24.614445 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:24.614521 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":156,"threads_started":1}
I20260928 16:42:25.150084 8810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:25.149472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:44127)
0928 16:42:25.149810 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:25.149891 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
I20260928 16:42:25.374931 8811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:25.374166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.66:50175)
0928 16:42:25.374672 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:25.374790 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":125,"threads_started":1}
I20260928 16:42:25.616869 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:25.615815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.67:58887)
0928 16:42:25.616617 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:25.616707 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":113,"threads_started":1}
I20260928 16:42:26.151834 8813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:26.151304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:53753)
0928 16:42:26.151631 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:26.151701 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":104,"threads_started":1}
I20260928 16:42:26.376909 8814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:26.376202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.66:45469)
0928 16:42:26.376649 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:26.376755 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":113,"threads_started":1}
I20260928 16:42:26.618542 8815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:26.618037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.67:52455)
0928 16:42:26.618371 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:26.618436 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":106,"threads_started":1}
I20260928 16:42:27.153640 8816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:27.152887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:47361)
0928 16:42:27.153325 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:27.153461 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":87,"threads_started":1}
I20260928 16:42:27.378813 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:27.378217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.66:43151)
0928 16:42:27.378571 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:27.378677 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":103,"threads_started":1}
I20260928 16:42:27.620324 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:27.619658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.67:58641)
0928 16:42:27.620102 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:27.620198 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":114,"threads_started":1}
I20260928 16:42:28.155530 8819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:28.154874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:44785)
0928 16:42:28.155281 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:28.155376 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":102,"threads_started":1}
I20260928 16:42:28.297341 8820 raft_consensus.cc:482] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:42:28.297560 8820 raft_consensus.cc:504] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbbadcc74d249c6aa3a29d6010cb055" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 34275 } } peers { permanent_uuid: "df42deab246142cbb05bff62c39fc80a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40907 } } peers { permanent_uuid: "ca982d23e01e43f3ac61669de9c27699" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41307 } }
I20260928 16:42:28.298157 8820 leader_election.cc:288] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cbbadcc74d249c6aa3a29d6010cb055 (127.25.71.67:34275), df42deab246142cbb05bff62c39fc80a (127.25.71.66:40907)
W20260928 16:42:28.299237 8395 leader_election.cc:341] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df42deab246142cbb05bff62c39fc80a (127.25.71.66:40907): Not found: Tablet not found: 7c2eff780a194fbf9b2ed88f4e5f79f5
W20260928 16:42:28.299377 8396 leader_election.cc:341] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cbbadcc74d249c6aa3a29d6010cb055 (127.25.71.67:34275): Not found: Tablet not found: 7c2eff780a194fbf9b2ed88f4e5f79f5
I20260928 16:42:28.299511 8396 leader_election.cc:302] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca982d23e01e43f3ac61669de9c27699; no voters: 0cbbadcc74d249c6aa3a29d6010cb055, df42deab246142cbb05bff62c39fc80a
I20260928 16:42:28.299780 8820 raft_consensus.cc:2726] T 7c2eff780a194fbf9b2ed88f4e5f79f5 P ca982d23e01e43f3ac61669de9c27699 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:42:28.380749 8821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:28.379984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.66:55691)
0928 16:42:28.380474 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:28.380606 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":86,"threads_started":1}
I20260928 16:42:28.622454 8822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:28.621777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.67:57631)
0928 16:42:28.622215 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:28.622344 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":118,"threads_started":1}
I20260928 16:42:29.157322 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:29.156650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:40085)
0928 16:42:29.157078 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:29.157164 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":133,"threads_started":1}
I20260928 16:42:29.382767 8824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:29.382183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.66:35603)
0928 16:42:29.382504 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:29.382573 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":82,"threads_started":1}
I20260928 16:42:29.624193 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:29.623527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.67:45383)
0928 16:42:29.623971 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:29.624056 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":98,"threads_started":1}
I20260928 16:42:30.159198 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:30.158531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:33551)
0928 16:42:30.159001 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:30.159086 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":173,"threads_started":1}
I20260928 16:42:30.384591 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:30.383955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.66:51123)
0928 16:42:30.384369 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:30.384457 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":124,"threads_started":1}
I20260928 16:42:30.625937 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:30.625280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.67:43537)
0928 16:42:30.625691 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:30.625796 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":97,"threads_started":1}
I20260928 16:42:31.161055 8829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:31.160443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.65:58437)
0928 16:42:31.160837 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:31.160912 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":115,"threads_started":1}
I20260928 16:42:31.386385 8830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:31.385826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35439 (local address 127.25.71.66:55965)
0928 16:42:31.386194 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:31.386264 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":122,"threads_started":1}
I20260928 16:42:31.602264 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 8344
I20260928 16:42:31.609823 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 8511
I20260928 16:42:31.616864 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 8646
2026-09-28T16:42:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19882 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:42:31 dist-test-slave-nqgh krb5kdc[8833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:42:31 dist-test-slave-nqgh krb5kdc[8833](info): set up 2 sockets
Sep 28 16:42:31 dist-test-slave-nqgh krb5kdc[8833](info): commencing operation
krb5kdc: starting...
W20260928 16:42:32.612675 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:42:32 dist-test-slave-nqgh krb5kdc[8833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:42:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:42:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:42:32.754458 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:37361
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:37361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:42:32.880196 8846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:32.880527 8846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:32.880571 8846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:32.884889 8846 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 16:42:32.884969 8846 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 16:42:32.884984 8846 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 16:42:32.884996 8846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:42:32.885015 8846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:32.885033 8846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:32.885048 8846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:42:32.885063 8846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:42:32.890858 8846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:37361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:37361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:32.892041 8846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:32.893088 8846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:32.903429 8851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:32.903470 8854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:32.903470 8852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:32.903719 8846 server_base.cc:1034] running on GCE node
I20260928 16:42:32.904450 8846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:32.905792 8846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:32.907217 8846 hybrid_clock.cc:648] HybridClock initialized: now 1790613752907178 us; error 60 us; skew 500 ppm
Sep 28 16:42:32 dist-test-slave-nqgh krb5kdc[8833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613752, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:32.910547 8846 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:42:32.911716 8846 webserver.cc:460] Webserver started at http://127.25.71.126:44169/ using document root <none> and password file <none>
I20260928 16:42:32.912331 8846 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:32.912415 8846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:32.912639 8846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:42:32.915017 8846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "56436701b863475f99bf93e6d98fadee"
format_stamp: "Formatted at 2026-09-28 16:42:32 on dist-test-slave-nqgh"
server_key: "4d9c43daffb57ecb6f42410e9709b4c5"
server_key_iv: "71e1f4d009986b959b066e02f112f737"
server_key_version: "encryptionkey@0"
I20260928 16:42:32.915620 8846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "56436701b863475f99bf93e6d98fadee"
format_stamp: "Formatted at 2026-09-28 16:42:32 on dist-test-slave-nqgh"
server_key: "4d9c43daffb57ecb6f42410e9709b4c5"
server_key_iv: "71e1f4d009986b959b066e02f112f737"
server_key_version: "encryptionkey@0"
I20260928 16:42:32.920519 8846 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 16:42:32.924510 8861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:32.926573 8846 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:42:32.926892 8846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "56436701b863475f99bf93e6d98fadee"
format_stamp: "Formatted at 2026-09-28 16:42:32 on dist-test-slave-nqgh"
server_key: "4d9c43daffb57ecb6f42410e9709b4c5"
server_key_iv: "71e1f4d009986b959b066e02f112f737"
server_key_version: "encryptionkey@0"
I20260928 16:42:32.927069 8846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:32.953191 8846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:32.957459 8846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:32.957710 8846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:32.967270 8846 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:37361
I20260928 16:42:32.967579 8913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:37361 every 8 connection(s)
I20260928 16:42:32.968700 8846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:42:32.973378 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 8846
I20260928 16:42:32.973533 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:42:32.973574 8914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:32.973867 25885 external_mini_cluster.cc:1428] Setting key 67b669f0d59f54e145686b24bd239eef
I20260928 16:42:32.980907 8914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee: Bootstrap starting.
I20260928 16:42:32.983793 8914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee: Neither blocks nor log segments found. Creating new log.
I20260928 16:42:32.984866 8914 log.cc:824] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:42:32 dist-test-slave-nqgh krb5kdc[8833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:32.988340 8914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee: No bootstrap required, opened a new log
I20260928 16:42:32.993733 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:32.976056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57426 (local address 127.25.71.126:37361)
0928 16:42:32.976645 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:32.976675 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:32.976721 (+ 46us) server_negotiation.cc:407] Connection header received
0928 16:42:32.977605 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:32.977630 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:32.978112 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:32.978515 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:32.979621 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:32.980868 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:32.982242 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:32.983007 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:32.983192 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:32.986770 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:32.986779 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:32.986804 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:32.986845 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:32.989297 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:32.989917 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:32.989924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:32.989932 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:32.990108 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:32.990480 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:32.990484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:32.990486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:32.991287 (+ 801us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:32.991560 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:32.991856 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 16:42:32.992304 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":170,"threads_started":1}
I20260928 16:42:32.994095 8914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56436701b863475f99bf93e6d98fadee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37361 } }
I20260928 16:42:32.994477 8914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:32.994539 8914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56436701b863475f99bf93e6d98fadee, State: Initialized, Role: FOLLOWER
I20260928 16:42:32.995258 8914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56436701b863475f99bf93e6d98fadee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37361 } }
I20260928 16:42:32.995458 8914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:42:32.995535 8914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:42:32.995656 8914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:32.997017 8914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56436701b863475f99bf93e6d98fadee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37361 } }
I20260928 16:42:32.997486 8914 leader_election.cc:302] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56436701b863475f99bf93e6d98fadee; no voters:
I20260928 16:42:32.997898 8914 leader_election.cc:288] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:42:32.998153 8919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:42:32.998451 8919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 1 LEADER]: Becoming Leader. State: Replica: 56436701b863475f99bf93e6d98fadee, State: Running, Role: LEADER
I20260928 16:42:32.998978 8919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56436701b863475f99bf93e6d98fadee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37361 } }
I20260928 16:42:32.999513 8914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:42:33.001101 8920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56436701b863475f99bf93e6d98fadee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56436701b863475f99bf93e6d98fadee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37361 } } }
I20260928 16:42:33.001042 8921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56436701b863475f99bf93e6d98fadee. Latest consensus state: current_term: 1 leader_uuid: "56436701b863475f99bf93e6d98fadee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56436701b863475f99bf93e6d98fadee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37361 } } }
I20260928 16:42:33.001312 8920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [sys.catalog]: This master's current role is: LEADER
I20260928 16:42:33.001327 8921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [sys.catalog]: This master's current role is: LEADER
I20260928 16:42:33.002873 8929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:42:33.007771 8929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:42:33.015470 8929 catalog_manager.cc:1347] Generated new cluster ID: 76d1f907ef46487dacb3e5e99b1831a8
I20260928 16:42:33.015605 8929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:42:33.024456 8929 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:42:33.025894 8929 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:42:33.033789 8929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee: Generated new TSK 0
I20260928 16:42:33.034885 8929 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
I20260928 16:42:33.079725 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:37361
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:42:33.225884 8940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:33.226313 8940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:33.226365 8940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:33.231179 8940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:33.231302 8940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:33.231376 8940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:42:33.237990 8940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:37361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:33.239624 8940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:33.241104 8940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:33.254817 8945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:33.254818 8946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:33.254863 8940 server_base.cc:1034] running on GCE node
W20260928 16:42:33.254969 8948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:33.257113 8940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:33.258210 8940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:33.259483 8940 hybrid_clock.cc:648] HybridClock initialized: now 1790613753259429 us; error 71 us; skew 500 ppm
Sep 28 16:42:33 dist-test-slave-nqgh krb5kdc[8833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613753, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:33.264361 8940 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:42:33.265977 8940 webserver.cc:460] Webserver started at http://127.25.71.65:36501/ using document root <none> and password file <none>
I20260928 16:42:33.266909 8940 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:33.267014 8940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:33.267259 8940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:42:33.270202 8940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "7cda6e131080426490efc2ba9323001e"
format_stamp: "Formatted at 2026-09-28 16:42:33 on dist-test-slave-nqgh"
server_key: "97fe083de4df83fed0d0b95a2a0ac00c"
server_key_iv: "65ae4e60d3db4c43e0c84473c6997e2f"
server_key_version: "encryptionkey@0"
I20260928 16:42:33.271045 8940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "7cda6e131080426490efc2ba9323001e"
format_stamp: "Formatted at 2026-09-28 16:42:33 on dist-test-slave-nqgh"
server_key: "97fe083de4df83fed0d0b95a2a0ac00c"
server_key_iv: "65ae4e60d3db4c43e0c84473c6997e2f"
server_key_version: "encryptionkey@0"
I20260928 16:42:33.276569 8940 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:42:33.281116 8955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:33.283444 8940 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260928 16:42:33.283681 8940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "7cda6e131080426490efc2ba9323001e"
format_stamp: "Formatted at 2026-09-28 16:42:33 on dist-test-slave-nqgh"
server_key: "97fe083de4df83fed0d0b95a2a0ac00c"
server_key_iv: "65ae4e60d3db4c43e0c84473c6997e2f"
server_key_version: "encryptionkey@0"
I20260928 16:42:33.283869 8940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:33.306959 8940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:33.311525 8940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:33.311766 8940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:33.312569 8940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:42:33.313687 8940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:42:33.313763 8940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:33.313817 8940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:42:33.313841 8940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:33.326256 8940 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:35451
I20260928 16:42:33.326303 9068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:35451 every 8 connection(s)
I20260928 16:42:33.327493 8940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:42:33.329908 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 8940
I20260928 16:42:33.330170 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:42:33.330595 25885 external_mini_cluster.cc:1428] Setting key bdd42217cef5a9d4fafa93700020ea26
Sep 28 16:42:33 dist-test-slave-nqgh krb5kdc[8833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613753, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.66@KRBTEST.COM" created.
I20260928 16:42:33.347182 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:33.330049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:38759 (local address 127.25.71.126:37361)
0928 16:42:33.330210 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:33.330214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:33.331911 (+ 1697us) server_negotiation.cc:407] Connection header received
0928 16:42:33.333852 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:33.333856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:33.333921 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:33.334122 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:33.335913 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.336502 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.337739 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.337998 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.338094 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:33.341557 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:33.341576 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:33.341580 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:33.341620 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:33.343489 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:33.344441 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:33.344445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:33.344446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:33.344559 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:33.345094 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:33.345097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:33.345098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:33.345388 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:33.345629 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:33.346758 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0928 16:42:33.346991 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:42:33.348356 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:33.330793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.65:38759)
0928 16:42:33.331699 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:33.331788 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:33.333445 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:33.334360 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:33.334371 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:33.335002 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:33.335709 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:33.335726 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.336645 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.336648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:33.337606 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:33.337617 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.338163 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.338167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:33.338349 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:33.338975 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:33.338993 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:33.341048 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:33.343742 (+ 2694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:33.343752 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:33.343776 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:33.344272 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:33.344730 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:33.344733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:33.344735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:33.344958 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:33.345646 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:33.345655 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:33.346076 (+ 421us) client_negotiation.cc:769] Sending connection context
0928 16:42:33.346458 (+ 382us) client_negotiation.cc:240] Negotiation successful
0928 16:42:33.347032 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":277,"threads_started":1}
I20260928 16:42:33.350414 9069 heartbeater.cc:347] Connected to a master server at 127.25.71.126:37361
I20260928 16:42:33.350847 9069 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:33.351655 9069 heartbeater.cc:511] Master 127.25.71.126:37361 requested a full tablet report, sending...
I20260928 16:42:33.354144 8877 ts_manager.cc:194] Registered new tserver with Master: 7cda6e131080426490efc2ba9323001e (127.25.71.65:35451)
I20260928 16:42:33.355774 8877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:38759
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
I20260928 16:42:33.368094 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:37361
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:42:33.489403 9074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:33.489706 9074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:33.489771 9074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:33.494033 9074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:33.494120 9074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:33.494205 9074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:42:33.500174 9074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:37361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:33.501475 9074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:33.502765 9074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:33.514001 9079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:33.514106 9074 server_base.cc:1034] running on GCE node
W20260928 16:42:33.513993 9082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:33.514127 9080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:33.514948 9074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:33.515867 9074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:33.517134 9074 hybrid_clock.cc:648] HybridClock initialized: now 1790613753517105 us; error 51 us; skew 500 ppm
Sep 28 16:42:33 dist-test-slave-nqgh krb5kdc[8833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613753, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:33.520905 9074 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:42:33.522324 9074 webserver.cc:460] Webserver started at http://127.25.71.66:39663/ using document root <none> and password file <none>
I20260928 16:42:33.523180 9074 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:33.523272 9074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:33.523538 9074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:42:33.526393 9074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "a3c79619df0b47b8a1700d9d207151f5"
format_stamp: "Formatted at 2026-09-28 16:42:33 on dist-test-slave-nqgh"
server_key: "8b6d9062e3534af17230afd9557bde8d"
server_key_iv: "f82ccfa990fce72a726c4a62920969d2"
server_key_version: "encryptionkey@0"
I20260928 16:42:33.527238 9074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "a3c79619df0b47b8a1700d9d207151f5"
format_stamp: "Formatted at 2026-09-28 16:42:33 on dist-test-slave-nqgh"
server_key: "8b6d9062e3534af17230afd9557bde8d"
server_key_iv: "f82ccfa990fce72a726c4a62920969d2"
server_key_version: "encryptionkey@0"
I20260928 16:42:33.532537 9074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 16:42:33.536370 9089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:33.538275 9074 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 16:42:33.538467 9074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "a3c79619df0b47b8a1700d9d207151f5"
format_stamp: "Formatted at 2026-09-28 16:42:33 on dist-test-slave-nqgh"
server_key: "8b6d9062e3534af17230afd9557bde8d"
server_key_iv: "f82ccfa990fce72a726c4a62920969d2"
server_key_version: "encryptionkey@0"
I20260928 16:42:33.538739 9074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:33.553085 9074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:33.557305 9074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:33.557520 9074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:33.558247 9074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:42:33.559389 9074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:42:33.559448 9074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:33.559499 9074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:42:33.559523 9074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:33.571230 9074 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:46519
I20260928 16:42:33.571376 9202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:46519 every 8 connection(s)
I20260928 16:42:33.572468 9074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:42:33.577587 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 9074
I20260928 16:42:33.577721 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:42:33.578028 25885 external_mini_cluster.cc:1428] Setting key a147ba48c97960db581a85f37f51f4a7
Sep 28 16:42:33 dist-test-slave-nqgh krb5kdc[8833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613753, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.71.67@KRBTEST.COM; defaulting to no policy
I20260928 16:42:33.592725 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:33.574972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:60557 (local address 127.25.71.126:37361)
0928 16:42:33.575274 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:33.575280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:33.576766 (+ 1486us) server_negotiation.cc:407] Connection header received
0928 16:42:33.578109 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:33.578113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:33.578192 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:33.578327 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:33.580585 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.581479 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.583759 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.584055 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.584193 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:33.587918 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:33.587924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:33.587927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:33.587977 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:33.589987 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:33.590767 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:33.590771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:33.590772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:33.590872 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:33.591275 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:33.591277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:33.591278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:33.591516 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:33.591611 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:33.592318 (+ 707us) server_negotiation.cc:299] Negotiation successful
0928 16:42:33.592502 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.67@KRBTEST.COM" created.
I20260928 16:42:33.593943 9205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:33.575359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.66:60557)
0928 16:42:33.576498 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:33.576582 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:33.577769 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:33.578697 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:33.578709 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:33.579448 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:33.580345 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:33.580373 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.581737 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.581741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:33.583518 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:33.583539 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.584295 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.584299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:33.584511 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:33.585303 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:33.585325 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:33.587518 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:33.590182 (+ 2664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:33.590192 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:33.590216 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:33.590605 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:33.590969 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:33.590972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:33.590975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:33.591169 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:33.591627 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:33.591636 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:33.592016 (+ 380us) client_negotiation.cc:769] Sending connection context
0928 16:42:33.592241 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 16:42:33.592714 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":602,"threads_started":1}
I20260928 16:42:33.596021 9203 heartbeater.cc:347] Connected to a master server at 127.25.71.126:37361
I20260928 16:42:33.596386 9203 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:33.597193 9203 heartbeater.cc:511] Master 127.25.71.126:37361 requested a full tablet report, sending...
I20260928 16:42:33.599126 8877 ts_manager.cc:194] Registered new tserver with Master: a3c79619df0b47b8a1700d9d207151f5 (127.25.71.66:46519)
I20260928 16:42:33.599828 8877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:60557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
I20260928 16:42:33.615875 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:37361
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:42:33.732234 9208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:33.732571 9208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:33.732618 9208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:33.736893 9208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:33.736986 9208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:33.737047 9208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:42:33.742787 9208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:37361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:33.743989 9208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:33.745080 9208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:33.756767 9216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:33.756815 9214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:33.756783 9213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:33.757138 9208 server_base.cc:1034] running on GCE node
I20260928 16:42:33.757656 9208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:33.758414 9208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:33.759658 9208 hybrid_clock.cc:648] HybridClock initialized: now 1790613753759615 us; error 67 us; skew 500 ppm
Sep 28 16:42:33 dist-test-slave-nqgh krb5kdc[8833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613753, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:33.762794 9208 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:42:33.763917 9208 webserver.cc:460] Webserver started at http://127.25.71.67:44519/ using document root <none> and password file <none>
I20260928 16:42:33.764705 9208 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:33.764796 9208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:33.764997 9208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:42:33.767406 9208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "05db9f8c71e84e3e954c351ee1a54f0a"
format_stamp: "Formatted at 2026-09-28 16:42:33 on dist-test-slave-nqgh"
server_key: "6c2dc4e8fb2353e1a489d0e175990ee9"
server_key_iv: "943be50e8343cacd99855807c854de4f"
server_key_version: "encryptionkey@0"
I20260928 16:42:33.768060 9208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "05db9f8c71e84e3e954c351ee1a54f0a"
format_stamp: "Formatted at 2026-09-28 16:42:33 on dist-test-slave-nqgh"
server_key: "6c2dc4e8fb2353e1a489d0e175990ee9"
server_key_iv: "943be50e8343cacd99855807c854de4f"
server_key_version: "encryptionkey@0"
I20260928 16:42:33.773331 9208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:42:33.776389 9223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:33.777962 9208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:42:33.778103 9208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "05db9f8c71e84e3e954c351ee1a54f0a"
format_stamp: "Formatted at 2026-09-28 16:42:33 on dist-test-slave-nqgh"
server_key: "6c2dc4e8fb2353e1a489d0e175990ee9"
server_key_iv: "943be50e8343cacd99855807c854de4f"
server_key_version: "encryptionkey@0"
I20260928 16:42:33.778242 9208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:33.797755 9208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:33.801621 9208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:33.801875 9208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:33.802569 9208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:42:33.803632 9208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:42:33.803687 9208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:33.803757 9208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:42:33.803784 9208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:33.814819 9208 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:41523
I20260928 16:42:33.814888 9336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:41523 every 8 connection(s)
I20260928 16:42:33.816064 9208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:42:33.824493 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 9208
I20260928 16:42:33.824647 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:42:33.824939 25885 external_mini_cluster.cc:1428] Setting key 4607eec2d10979cb8ea3facb5fb324c3
Sep 28 16:42:33 dist-test-slave-nqgh krb5kdc[8833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613753, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:33.834899 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:33.818533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:35917 (local address 127.25.71.126:37361)
0928 16:42:33.818778 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:33.818784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:33.820083 (+ 1299us) server_negotiation.cc:407] Connection header received
0928 16:42:33.821686 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:33.821689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:33.821769 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:33.821895 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:33.823437 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.824175 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.826125 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.826505 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.826658 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:33.830240 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:33.830244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:33.830246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:33.830283 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:33.831933 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:33.832858 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:33.832862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:33.832863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:33.832985 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:33.833444 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:33.833446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:33.833447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:33.833670 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:33.833755 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:33.834560 (+ 805us) server_negotiation.cc:299] Negotiation successful
0928 16:42:33.834750 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260928 16:42:33.836108 9339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:33.819107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.67:35917)
0928 16:42:33.819849 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:33.819921 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:33.821323 (+ 1402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:33.822101 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:33.822109 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:33.822561 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:33.823215 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:33.823230 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.824387 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.824391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:33.825895 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:33.825918 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.826758 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.826765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:33.827023 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:33.827899 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:33.827926 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:33.829916 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:33.832241 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:33.832252 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:33.832274 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:33.832702 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:33.833119 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:33.833123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:33.833126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:33.833336 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:33.833798 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:33.833806 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:33.834215 (+ 409us) client_negotiation.cc:769] Sending connection context
0928 16:42:33.834428 (+ 213us) client_negotiation.cc:240] Negotiation successful
0928 16:42:33.834879 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":181,"threads_started":1}
I20260928 16:42:33.837877 9337 heartbeater.cc:347] Connected to a master server at 127.25.71.126:37361
I20260928 16:42:33.838191 9337 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:33.838932 9337 heartbeater.cc:511] Master 127.25.71.126:37361 requested a full tablet report, sending...
I20260928 16:42:33.840368 8877 ts_manager.cc:194] Registered new tserver with Master: 05db9f8c71e84e3e954c351ee1a54f0a (127.25.71.67:41523)
I20260928 16:42:33.840950 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:42:33.841231 8877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:35917
I20260928 16:42:33.852571 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:33.844171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57436 (local address 127.25.71.126:37361)
0928 16:42:33.844322 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:33.844327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:33.844560 (+ 233us) server_negotiation.cc:407] Connection header received
0928 16:42:33.844781 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:33.844784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:33.844840 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:33.844913 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:33.845790 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.846394 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.847348 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.847594 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.847691 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:33.849265 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:33.849269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:33.849272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:33.849301 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:33.850780 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:33.851213 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:33.851217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:33.851219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:33.851304 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:33.851554 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:33.851558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:33.851560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:33.851858 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:33.851995 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:33.852237 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 16:42:33.852378 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:42:33.856627 8877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:42:33.860010 8877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:42:33.878470 9349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:33.871340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:41523 (local address 127.0.0.1:41156)
0928 16:42:33.872533 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:33.872613 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:33.872796 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:33.873468 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:33.873475 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:33.873498 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:33.873662 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:33.873670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.874917 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.874921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:33.876302 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:33.876312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.877684 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.877688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:33.877851 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:33.877901 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 16:42:33.878031 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 16:42:33.878256 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"mutex_wait_us":60,"thread_start_us":107,"threads_started":1}
I20260928 16:42:33.878559 9353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:33.872307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41156 (local address 127.25.71.67:41523)
0928 16:42:33.872731 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:33.872741 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:33.872767 (+ 26us) server_negotiation.cc:407] Connection header received
0928 16:42:33.873047 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:33.873061 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:33.873275 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:33.873457 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:33.873797 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.874747 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.876523 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.877512 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.877656 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:33.877843 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:33.878215 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 16:42:33.878381 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":142,"threads_started":1}
I20260928 16:42:33.878989 9352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:33.871462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:35451 (local address 127.0.0.1:54396)
0928 16:42:33.872915 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:33.872929 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:33.873008 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:33.873639 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:33.873641 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:33.873655 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:33.873733 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:33.873740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.875212 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.875216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:33.876797 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:33.876806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.877869 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.877872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:33.877948 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:33.877967 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 16:42:33.878021 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 16:42:33.878192 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1396,"mutex_wait_us":118,"thread_start_us":686,"threads_started":1}
I20260928 16:42:33.879179 9350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:33.871763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54396 (local address 127.25.71.65:35451)
0928 16:42:33.872540 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:33.872547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:33.872998 (+ 451us) server_negotiation.cc:407] Connection header received
0928 16:42:33.873157 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:33.873162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:33.873346 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:33.873493 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:33.873842 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.875068 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.876947 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.877548 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.877645 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:33.877811 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:33.878362 (+ 551us) server_negotiation.cc:299] Negotiation successful
0928 16:42:33.878459 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":118,"threads_started":1}
I20260928 16:42:33.879630 9351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:33.871819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40406 (local address 127.25.71.66:46519)
0928 16:42:33.872793 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:33.872808 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:33.872837 (+ 29us) server_negotiation.cc:407] Connection header received
0928 16:42:33.873005 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:33.873011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:33.873276 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:33.873457 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:33.874045 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.875224 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.876970 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.877847 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.877963 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:33.878112 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:33.879333 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0928 16:42:33.879474 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849,"thread_start_us":169,"threads_started":1}
I20260928 16:42:33.879961 9348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:33.871383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:46519 (local address 127.0.0.1:40406)
0928 16:42:33.872691 (+ 1308us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:33.872707 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:33.872823 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:33.873686 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:33.873689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:33.873708 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:33.873865 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:33.873873 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.875375 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.875378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:33.876793 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:33.876806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:33.877981 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:33.877986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:33.878126 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:33.878155 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 16:42:33.878228 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 16:42:33.878306 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1227,"mutex_wait_us":126,"thread_start_us":223,"threads_started":1}
I20260928 16:42:33.882045 9003 tablet_service.cc:1467] Processing CreateTablet for tablet 5922457dc098434a8591749a0d3e2f31 (DEFAULT_TABLE table=test-table [id=0708d47c61d04cdb9f7a51255d907c24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:42:33.882333 9271 tablet_service.cc:1467] Processing CreateTablet for tablet 5922457dc098434a8591749a0d3e2f31 (DEFAULT_TABLE table=test-table [id=0708d47c61d04cdb9f7a51255d907c24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:42:33.883320 9003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5922457dc098434a8591749a0d3e2f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:33.883739 9271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5922457dc098434a8591749a0d3e2f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:33.885841 9137 tablet_service.cc:1467] Processing CreateTablet for tablet 5922457dc098434a8591749a0d3e2f31 (DEFAULT_TABLE table=test-table [id=0708d47c61d04cdb9f7a51255d907c24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:42:33.887300 9137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5922457dc098434a8591749a0d3e2f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:33.890659 9354 tablet_bootstrap.cc:492] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e: Bootstrap starting.
I20260928 16:42:33.892633 9355 tablet_bootstrap.cc:492] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a: Bootstrap starting.
I20260928 16:42:33.893406 9354 tablet_bootstrap.cc:654] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e: Neither blocks nor log segments found. Creating new log.
I20260928 16:42:33.894544 9354 log.cc:824] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:33.895323 9355 tablet_bootstrap.cc:654] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a: Neither blocks nor log segments found. Creating new log.
I20260928 16:42:33.896317 9357 tablet_bootstrap.cc:492] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5: Bootstrap starting.
I20260928 16:42:33.896860 9355 log.cc:824] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:33.897378 9354 tablet_bootstrap.cc:492] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e: No bootstrap required, opened a new log
I20260928 16:42:33.897639 9354 ts_tablet_manager.cc:1397] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260928 16:42:33.899606 9355 tablet_bootstrap.cc:492] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a: No bootstrap required, opened a new log
I20260928 16:42:33.899696 9357 tablet_bootstrap.cc:654] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5: Neither blocks nor log segments found. Creating new log.
I20260928 16:42:33.900010 9355 ts_tablet_manager.cc:1397] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 16:42:33.900959 9357 log.cc:824] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:33.902453 9354 raft_consensus.cc:348] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:42:33.902876 9354 raft_consensus.cc:374] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:33.902948 9354 raft_consensus.cc:729] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cda6e131080426490efc2ba9323001e, State: Initialized, Role: FOLLOWER
I20260928 16:42:33.903523 9354 consensus_queue.cc:260] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:42:33.904043 9355 raft_consensus.cc:348] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:42:33.904335 9355 raft_consensus.cc:374] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:33.904405 9355 raft_consensus.cc:729] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05db9f8c71e84e3e954c351ee1a54f0a, State: Initialized, Role: FOLLOWER
I20260928 16:42:33.904454 9069 heartbeater.cc:503] Master 127.25.71.126:37361 was elected leader, sending a full tablet report...
I20260928 16:42:33.904860 9355 consensus_queue.cc:260] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:42:33.905165 9357 tablet_bootstrap.cc:492] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5: No bootstrap required, opened a new log
I20260928 16:42:33.905452 9357 ts_tablet_manager.cc:1397] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260928 16:42:33.905818 9337 heartbeater.cc:503] Master 127.25.71.126:37361 was elected leader, sending a full tablet report...
I20260928 16:42:33.905953 9355 ts_tablet_manager.cc:1428] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:42:33.907053 9354 ts_tablet_manager.cc:1428] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260928 16:42:33.911157 9357 raft_consensus.cc:348] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:42:33.911517 9357 raft_consensus.cc:374] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:33.911573 9357 raft_consensus.cc:729] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c79619df0b47b8a1700d9d207151f5, State: Initialized, Role: FOLLOWER
I20260928 16:42:33.912074 9357 consensus_queue.cc:260] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:42:33.912912 9203 heartbeater.cc:503] Master 127.25.71.126:37361 was elected leader, sending a full tablet report...
I20260928 16:42:33.913237 9357 ts_tablet_manager.cc:1428] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
W20260928 16:42:34.067260 9338 tablet.cc:2369] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:42:34.074156 9204 tablet.cc:2369] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:42:34.079506 9070 tablet.cc:2369] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:42:34.084443 9360 raft_consensus.cc:482] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:42:34.084811 9360 raft_consensus.cc:504] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:42:34.086369 9360 leader_election.cc:288] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3c79619df0b47b8a1700d9d207151f5 (127.25.71.66:46519), 05db9f8c71e84e3e954c351ee1a54f0a (127.25.71.67:41523)
I20260928 16:42:34.091679 9353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:34.087007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:37663 (local address 127.25.71.67:41523)
0928 16:42:34.087349 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:34.087354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:34.087515 (+ 161us) server_negotiation.cc:407] Connection header received
0928 16:42:34.087705 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:34.087710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:34.087765 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:34.087905 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:34.088307 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:34.088926 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:34.090516 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:34.090956 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:34.091073 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:34.091121 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:34.091441 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 16:42:34.091530 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260928 16:42:34.091805 9351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:34.087007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:34411 (local address 127.25.71.66:46519)
0928 16:42:34.087187 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:34.087194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:34.087647 (+ 453us) server_negotiation.cc:407] Connection header received
0928 16:42:34.087864 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:34.087868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:34.087936 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:34.088042 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:34.088436 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:34.089083 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:34.090543 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:34.091066 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:34.091206 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:34.091276 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:34.091582 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 16:42:34.091680 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:42:34.091786 9363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:34.086840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:41523 (local address 127.25.71.65:37663)
0928 16:42:34.087375 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:34.087391 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:34.087567 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:34.087905 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:34.087909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:34.087931 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:34.088153 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:34.088163 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:34.089137 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:34.089141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:34.090360 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:34.090373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:34.091107 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:34.091111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:34.091288 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:34.091318 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 16:42:34.091393 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 16:42:34.091485 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":35,"thread_start_us":147,"threads_started":1}
I20260928 16:42:34.091799 9364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:34.086848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:46519 (local address 127.25.71.65:34411)
0928 16:42:34.087515 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:34.087533 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:34.087684 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:34.088079 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:34.088081 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:34.088101 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:34.088283 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:34.088292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:34.089227 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:34.089230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:34.090374 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:34.090386 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:34.091215 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:34.091218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:34.091414 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:34.091451 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 16:42:34.091534 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 16:42:34.091630 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":115,"thread_start_us":117,"threads_started":1}
I20260928 16:42:34.092753 9291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5922457dc098434a8591749a0d3e2f31" candidate_uuid: "7cda6e131080426490efc2ba9323001e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" is_pre_election: true
I20260928 16:42:34.092775 9157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5922457dc098434a8591749a0d3e2f31" candidate_uuid: "7cda6e131080426490efc2ba9323001e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c79619df0b47b8a1700d9d207151f5" is_pre_election: true
I20260928 16:42:34.093094 9291 raft_consensus.cc:2454] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cda6e131080426490efc2ba9323001e in term 0.
I20260928 16:42:34.093171 9157 raft_consensus.cc:2454] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cda6e131080426490efc2ba9323001e in term 0.
I20260928 16:42:34.093715 8958 leader_election.cc:302] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05db9f8c71e84e3e954c351ee1a54f0a, 7cda6e131080426490efc2ba9323001e, a3c79619df0b47b8a1700d9d207151f5; no voters:
I20260928 16:42:34.094067 9360 raft_consensus.cc:2781] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:42:34.094167 9360 raft_consensus.cc:482] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:42:34.094213 9360 raft_consensus.cc:3037] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:34.095371 9360 raft_consensus.cc:504] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:42:34.095811 9360 leader_election.cc:288] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 1 election: Requested vote from peers a3c79619df0b47b8a1700d9d207151f5 (127.25.71.66:46519), 05db9f8c71e84e3e954c351ee1a54f0a (127.25.71.67:41523)
I20260928 16:42:34.096393 9157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5922457dc098434a8591749a0d3e2f31" candidate_uuid: "7cda6e131080426490efc2ba9323001e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c79619df0b47b8a1700d9d207151f5"
I20260928 16:42:34.096478 9291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5922457dc098434a8591749a0d3e2f31" candidate_uuid: "7cda6e131080426490efc2ba9323001e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05db9f8c71e84e3e954c351ee1a54f0a"
I20260928 16:42:34.096611 9291 raft_consensus.cc:3037] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:34.096598 9157 raft_consensus.cc:3037] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:34.097813 9291 raft_consensus.cc:2454] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cda6e131080426490efc2ba9323001e in term 1.
I20260928 16:42:34.098274 9157 raft_consensus.cc:2454] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cda6e131080426490efc2ba9323001e in term 1.
I20260928 16:42:34.098306 8958 leader_election.cc:302] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05db9f8c71e84e3e954c351ee1a54f0a, 7cda6e131080426490efc2ba9323001e; no voters:
I20260928 16:42:34.098565 9360 raft_consensus.cc:2781] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:42:34.099126 9360 raft_consensus.cc:686] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [term 1 LEADER]: Becoming Leader. State: Replica: 7cda6e131080426490efc2ba9323001e, State: Running, Role: LEADER
I20260928 16:42:34.099570 9360 consensus_queue.cc:237] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:42:34.103492 8874 catalog_manager.cc:5495] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cda6e131080426490efc2ba9323001e (127.25.71.65). New cstate: current_term: 1 leader_uuid: "7cda6e131080426490efc2ba9323001e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } health_report { overall_health: HEALTHY } } }
I20260928 16:42:34.121395 9350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:34.117396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.25.71.65:35451)
0928 16:42:34.117589 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:34.117594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:34.117703 (+ 109us) server_negotiation.cc:407] Connection header received
0928 16:42:34.117942 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:34.117944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:34.118016 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:34.118119 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:42:34.118378 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:34.118942 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:34.119830 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:34.120047 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:34.120135 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:34.120480 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:42:34.120989 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:42:34.121101 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:34.121178 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 16:42:34.121242 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 16:42:34.129913 9291 raft_consensus.cc:1261] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a [term 1 FOLLOWER]: Refusing update from remote peer 7cda6e131080426490efc2ba9323001e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:42:34.130049 9157 raft_consensus.cc:1261] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5 [term 1 FOLLOWER]: Refusing update from remote peer 7cda6e131080426490efc2ba9323001e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:42:34.130853 9360 consensus_queue.cc:1035] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:42:34.131330 9365 consensus_queue.cc:1035] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:42:34.151358 8874 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:42:34.151584 8874 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:42:34.154704 8874 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee: Sending DeleteTablet for 3 replicas of tablet 5922457dc098434a8591749a0d3e2f31
I20260928 16:42:34.155732 9003 tablet_service.cc:1514] Processing DeleteTablet for tablet 5922457dc098434a8591749a0d3e2f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:42:34 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:54396
I20260928 16:42:34.155732 9137 tablet_service.cc:1514] Processing DeleteTablet for tablet 5922457dc098434a8591749a0d3e2f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:42:34 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:40406
I20260928 16:42:34.156008 9271 tablet_service.cc:1514] Processing DeleteTablet for tablet 5922457dc098434a8591749a0d3e2f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:42:34 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:41156
I20260928 16:42:34.156687 9378 tablet_replica.cc:330] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e: stopping tablet replica
I20260928 16:42:34.156839 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 8940
I20260928 16:42:34.157073 9378 raft_consensus.cc:2229] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:42:34.156852 9377 tablet_replica.cc:330] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5: stopping tablet replica
I20260928 16:42:34.157939 9377 raft_consensus.cc:2229] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:42:34.158459 9377 raft_consensus.cc:2258] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:42:34.158802 9379 tablet_replica.cc:330] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a: stopping tablet replica
I20260928 16:42:34.159237 9379 raft_consensus.cc:2229] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:42:34.159610 9379 raft_consensus.cc:2258] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:42:34.160821 9377 ts_tablet_manager.cc:1905] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:42:34.161077 9379 ts_tablet_manager.cc:1905] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:42:34.163523 9379 ts_tablet_manager.cc:1918] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:42:34.163627 9379 log.cc:1196] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/5922457dc098434a8591749a0d3e2f31
I20260928 16:42:34.163919 9379 ts_tablet_manager.cc:1939] T 5922457dc098434a8591749a0d3e2f31 P 05db9f8c71e84e3e954c351ee1a54f0a: Deleting consensus metadata
I20260928 16:42:34.164425 9377 ts_tablet_manager.cc:1918] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:42:34.164585 9377 log.cc:1196] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/5922457dc098434a8591749a0d3e2f31
I20260928 16:42:34.164901 9377 ts_tablet_manager.cc:1939] T 5922457dc098434a8591749a0d3e2f31 P a3c79619df0b47b8a1700d9d207151f5: Deleting consensus metadata
I20260928 16:42:34.165304 8864 catalog_manager.cc:4841] TS 05db9f8c71e84e3e954c351ee1a54f0a (127.25.71.67:41523): tablet 5922457dc098434a8591749a0d3e2f31 (table test-table [id=0708d47c61d04cdb9f7a51255d907c24]) successfully deleted
I20260928 16:42:34.166584 8862 catalog_manager.cc:4841] TS a3c79619df0b47b8a1700d9d207151f5 (127.25.71.66:46519): tablet 5922457dc098434a8591749a0d3e2f31 (table test-table [id=0708d47c61d04cdb9f7a51255d907c24]) successfully deleted
W20260928 16:42:34.167222 8863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.71.65:35451 (error 108)
I20260928 16:42:34.167503 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 9074
I20260928 16:42:34.169566 9348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:34.169267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:35451 (local address 127.0.0.1:54414)
0928 16:42:34.169413 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:34.169483 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.65:35451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 16:42:34.169732 8863 catalog_manager.cc:4584] TS 7cda6e131080426490efc2ba9323001e (127.25.71.65:35451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5922457dc098434a8591749a0d3e2f31: Network error: Client connection negotiation failed: client connection to 127.25.71.65:35451: connect: Connection refused (error 111)
I20260928 16:42:34.174521 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 9208
I20260928 16:42:34.181102 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 8846
I20260928 16:42:54.188323 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:37361
--webserver_interface=127.25.71.126
--webserver_port=44169
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:37361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:42:54.313687 9381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:54.313997 9381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:54.314041 9381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:54.318440 9381 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 16:42:54.318526 9381 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 16:42:54.318543 9381 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 16:42:54.318555 9381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:42:54.318573 9381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:54.318591 9381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:54.318605 9381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:42:54.318661 9381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:42:54.326929 9381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:37361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:37361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=44169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:54.328725 9381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:54.330550 9381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:42:54.349292 9381 server_base.cc:1034] running on GCE node
W20260928 16:42:54.349339 9387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:54.349386 9390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:54.349268 9388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:54.350579 9381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:54.352104 9381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:54.355485 9381 hybrid_clock.cc:648] HybridClock initialized: now 1790613774355437 us; error 60 us; skew 500 ppm
Sep 28 16:42:54 dist-test-slave-nqgh krb5kdc[8833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613774, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:54.359331 9381 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:42:54.360628 9381 webserver.cc:460] Webserver started at http://127.25.71.126:44169/ using document root <none> and password file <none>
I20260928 16:42:54.361292 9381 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:54.361392 9381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:54.366362 9381 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:42:54.369787 9397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:54.371968 9381 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:42:54.372226 9381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "56436701b863475f99bf93e6d98fadee"
format_stamp: "Formatted at 2026-09-28 16:42:32 on dist-test-slave-nqgh"
server_key: "4d9c43daffb57ecb6f42410e9709b4c5"
server_key_iv: "71e1f4d009986b959b066e02f112f737"
server_key_version: "encryptionkey@0"
I20260928 16:42:54.373047 9381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:54.390925 9381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:54.394888 9381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:54.395107 9381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:54.403671 9381 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:37361
I20260928 16:42:54.403760 9449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:37361 every 8 connection(s)
I20260928 16:42:54.405050 9381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:42:54.407783 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 9381
I20260928 16:42:54.408427 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:35451
--local_ip_for_outbound_sockets=127.25.71.65
--tserver_master_addrs=127.25.71.126:37361
--webserver_port=36501
--webserver_interface=127.25.71.65
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:42:54.413394 9450 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:42:54.416570 9450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee: Bootstrap starting.
I20260928 16:42:54.437561 9450 log.cc:824] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:54.447849 9450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:42:54.448292 9450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee: Bootstrap complete.
I20260928 16:42:54.452894 9450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56436701b863475f99bf93e6d98fadee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37361 } }
I20260928 16:42:54.453441 9450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56436701b863475f99bf93e6d98fadee, State: Initialized, Role: FOLLOWER
I20260928 16:42:54.454053 9450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56436701b863475f99bf93e6d98fadee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37361 } }
I20260928 16:42:54.454197 9450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:42:54.454257 9450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:42:54.454373 9450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:42:54.456874 9450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56436701b863475f99bf93e6d98fadee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37361 } }
I20260928 16:42:54.457283 9450 leader_election.cc:302] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56436701b863475f99bf93e6d98fadee; no voters:
I20260928 16:42:54.457628 9450 leader_election.cc:288] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 16:42:54.457746 9454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:42:54.457973 9454 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 2 LEADER]: Becoming Leader. State: Replica: 56436701b863475f99bf93e6d98fadee, State: Running, Role: LEADER
I20260928 16:42:54.458329 9454 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56436701b863475f99bf93e6d98fadee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37361 } }
I20260928 16:42:54.459168 9450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:42:54.460184 9455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "56436701b863475f99bf93e6d98fadee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56436701b863475f99bf93e6d98fadee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37361 } } }
I20260928 16:42:54.460434 9455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [sys.catalog]: This master's current role is: LEADER
I20260928 16:42:54.461028 9456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56436701b863475f99bf93e6d98fadee. Latest consensus state: current_term: 2 leader_uuid: "56436701b863475f99bf93e6d98fadee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56436701b863475f99bf93e6d98fadee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37361 } } }
I20260928 16:42:54.461239 9456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [sys.catalog]: This master's current role is: LEADER
I20260928 16:42:54.461170 9464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:42:54.469931 9464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:42:54.470978 9464 catalog_manager.cc:1259] Loaded cluster ID: 76d1f907ef46487dacb3e5e99b1831a8
I20260928 16:42:54.471077 9464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:42:54.474105 9464 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:42:54.475281 9464 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee: Loaded TSK: 0
I20260928 16:42:54.494586 9464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee: Generated new TSK 1
I20260928 16:42:54.495070 9464 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:42:54.560439 9452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:54.560815 9452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:54.560880 9452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:54.565547 9452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:54.565661 9452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:54.565734 9452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:42:54.572160 9452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:35451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=36501
--tserver_master_addrs=127.25.71.126:37361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:54.573721 9452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:54.575163 9452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:54.587571 9477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:54.587571 9478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:54.588167 9452 server_base.cc:1034] running on GCE node
W20260928 16:42:54.587678 9480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:54.588894 9452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:54.589751 9452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:54.591090 9452 hybrid_clock.cc:648] HybridClock initialized: now 1790613774591047 us; error 63 us; skew 500 ppm
Sep 28 16:42:54 dist-test-slave-nqgh krb5kdc[8833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613774, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:54.595086 9452 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:42:54.596519 9452 webserver.cc:460] Webserver started at http://127.25.71.65:36501/ using document root <none> and password file <none>
I20260928 16:42:54.597321 9452 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:54.597429 9452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:54.603063 9452 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:42:54.605937 9487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:54.607952 9452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:42:54.608136 9452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "7cda6e131080426490efc2ba9323001e"
format_stamp: "Formatted at 2026-09-28 16:42:33 on dist-test-slave-nqgh"
server_key: "97fe083de4df83fed0d0b95a2a0ac00c"
server_key_iv: "65ae4e60d3db4c43e0c84473c6997e2f"
server_key_version: "encryptionkey@0"
I20260928 16:42:54.608913 9452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:54.631482 9452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:54.635703 9452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:54.635915 9452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:54.636651 9452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:42:54.638072 9494 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 16:42:54.643258 9452 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 16:42:54.643404 9452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260928 16:42:54.643486 9452 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 16:42:54.646426 9452 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 16:42:54.646516 9452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260928 16:42:54.647092 9494 tablet_bootstrap.cc:492] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e: Bootstrap starting.
I20260928 16:42:54.659830 9452 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:35451
I20260928 16:42:54.659960 9601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:35451 every 8 connection(s)
I20260928 16:42:54.661289 9452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:42:54.668546 9494 log.cc:824] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:42:54.668643 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 9452
I20260928 16:42:54.669250 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:46519
--local_ip_for_outbound_sockets=127.25.71.66
--tserver_master_addrs=127.25.71.126:37361
--webserver_port=39663
--webserver_interface=127.25.71.66
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:42:54 dist-test-slave-nqgh krb5kdc[8833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613774, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:54.678586 9494 tablet_bootstrap.cc:492] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:42:54.679061 9494 tablet_bootstrap.cc:492] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e: Bootstrap complete.
I20260928 16:42:54.680334 9494 ts_tablet_manager.cc:1397] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e: Time spent bootstrapping tablet: real 0.034s user 0.024s sys 0.007s
I20260928 16:42:54.684087 9494 raft_consensus.cc:348] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:42:54.684813 9494 raft_consensus.cc:729] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cda6e131080426490efc2ba9323001e, State: Initialized, Role: FOLLOWER
I20260928 16:42:54.685587 9494 consensus_queue.cc:260] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:42:54.687206 9494 ts_tablet_manager.cc:1428] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 16:42:54.688618 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:54.664278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:47787 (local address 127.25.71.126:37361)
0928 16:42:54.664896 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:54.664923 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:54.665353 (+ 430us) server_negotiation.cc:407] Connection header received
0928 16:42:54.666914 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:54.666937 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:54.667470 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:54.667939 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:54.670361 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:54.672067 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:54.674055 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:54.674939 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:54.675162 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:54.679550 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:54.679557 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:54.679579 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:54.679627 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:54.682044 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:54.683186 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:54.683195 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:54.683202 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:54.683418 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:54.684111 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:54.684117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:54.684119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:54.685119 (+ 1000us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:54.685393 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:54.686768 (+ 1375us) server_negotiation.cc:299] Negotiation successful
0928 16:42:54.687347 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":189,"threads_started":1}
I20260928 16:42:54.689410 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:54.664090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.65:47787)
0928 16:42:54.665093 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:54.665167 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:54.666215 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:54.668212 (+ 1997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:54.668226 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:54.668967 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:54.669936 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:54.669973 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:54.672451 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:54.672458 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:54.673799 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:54.673823 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:54.675232 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:54.675238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:54.675589 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:54.676562 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:54.676593 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:54.679170 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:54.682293 (+ 3123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:54.682304 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:54.682333 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:54.682942 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:54.683611 (+ 669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:54.683639 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:54.683641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:54.683927 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:54.685466 (+ 1539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:54.685474 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:54.686047 (+ 573us) client_negotiation.cc:769] Sending connection context
0928 16:42:54.686315 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 16:42:54.688428 (+ 2113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":236,"threads_started":1}
I20260928 16:42:54.692993 9602 heartbeater.cc:347] Connected to a master server at 127.25.71.126:37361
I20260928 16:42:54.693385 9602 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:54.694191 9602 heartbeater.cc:511] Master 127.25.71.126:37361 requested a full tablet report, sending...
I20260928 16:42:54.697791 9410 ts_manager.cc:194] Registered new tserver with Master: 7cda6e131080426490efc2ba9323001e (127.25.71.65:35451)
I20260928 16:42:54.701843 9410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:47787
I20260928 16:42:54.705416 9602 heartbeater.cc:503] Master 127.25.71.126:37361 was elected leader, sending a full tablet report...
Sep 28 16:42:54 dist-test-slave-nqgh krb5kdc[8833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613774, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for kudu/127.25.71.65@KRBTEST.COM
I20260928 16:42:54.716341 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:54.701507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56564 (local address 127.25.71.65:35451)
0928 16:42:54.702089 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:54.702103 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:54.702771 (+ 668us) server_negotiation.cc:407] Connection header received
0928 16:42:54.703037 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:54.703045 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:54.703321 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:54.703543 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:54.704483 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:54.705686 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:54.707021 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:54.707585 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:54.707725 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:54.710596 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:54.710616 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:54.710669 (+ 53us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:54.710718 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:54.713101 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:54.714136 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:54.714146 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:54.714148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:54.714484 (+ 336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:54.714943 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:54.714946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:54.714947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:54.715299 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:54.715476 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:54.715977 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 16:42:54.716188 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":446,"thread_start_us":160,"threads_started":1}
I20260928 16:42:54.716373 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:54.701224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:35451 (local address 127.0.0.1:56564)
0928 16:42:54.702524 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:54.702580 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:54.702831 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:54.703593 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:54.703600 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:54.704141 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:54.704310 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:54.704319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:54.705864 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:54.705868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:54.706873 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:54.706884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:54.707743 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:54.707747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:54.707873 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:54.708686 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:54.708699 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:54.710338 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:54.713380 (+ 3042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:54.713402 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:54.713409 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:54.713889 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:54.714670 (+ 781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:54.714675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:54.714677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:54.714808 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:54.715453 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:54.715460 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:54.715662 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 16:42:54.715824 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 16:42:54.716128 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"thread_start_us":154,"threads_started":1}
I20260928 16:42:54.718094 9536 tablet_service.cc:1514] Processing DeleteTablet for tablet 5922457dc098434a8591749a0d3e2f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:42:34 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:56564
I20260928 16:42:54.719514 9611 tablet_replica.cc:330] stopping tablet replica
I20260928 16:42:54.719990 9611 raft_consensus.cc:2229] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:42:54.720281 9611 raft_consensus.cc:2258] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:42:54.721639 9611 ts_tablet_manager.cc:1905] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:42:54.726161 9611 ts_tablet_manager.cc:1918] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:42:54.726289 9611 log.cc:1196] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/wal/wals/5922457dc098434a8591749a0d3e2f31
I20260928 16:42:54.726547 9611 ts_tablet_manager.cc:1939] T 5922457dc098434a8591749a0d3e2f31 P 7cda6e131080426490efc2ba9323001e: Deleting consensus metadata
I20260928 16:42:54.729331 9399 catalog_manager.cc:4841] TS 7cda6e131080426490efc2ba9323001e (127.25.71.65:35451): tablet 5922457dc098434a8591749a0d3e2f31 (table test-table [id=0708d47c61d04cdb9f7a51255d907c24]) successfully deleted
W20260928 16:42:54.822208 9607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:54.822615 9607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:54.822708 9607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:54.827445 9607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:54.827567 9607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:54.827651 9607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:42:54.834476 9607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:46519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=39663
--tserver_master_addrs=127.25.71.126:37361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:54.836148 9607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:54.837538 9607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:54.850903 9617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:54.850960 9616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:54.851038 9607 server_base.cc:1034] running on GCE node
W20260928 16:42:54.850960 9619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:54.851922 9607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:54.852909 9607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:54.854202 9607 hybrid_clock.cc:648] HybridClock initialized: now 1790613774854143 us; error 78 us; skew 500 ppm
Sep 28 16:42:54 dist-test-slave-nqgh krb5kdc[8833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613774, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:54.858233 9607 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:42:54.859714 9607 webserver.cc:460] Webserver started at http://127.25.71.66:39663/ using document root <none> and password file <none>
I20260928 16:42:54.860553 9607 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:54.860661 9607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:54.866307 9607 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:42:54.869880 9626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:54.872080 9607 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:42:54.872323 9607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "a3c79619df0b47b8a1700d9d207151f5"
format_stamp: "Formatted at 2026-09-28 16:42:33 on dist-test-slave-nqgh"
server_key: "8b6d9062e3534af17230afd9557bde8d"
server_key_iv: "f82ccfa990fce72a726c4a62920969d2"
server_key_version: "encryptionkey@0"
I20260928 16:42:54.873158 9607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:54.893373 9607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:54.898049 9607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:54.898411 9607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:54.899372 9607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:42:54.900547 9607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:42:54.900610 9607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:54.900692 9607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:42:54.900718 9607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:54.912413 9607 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:46519
I20260928 16:42:54.912545 9739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:46519 every 8 connection(s)
I20260928 16:42:54.914078 9607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:42:54.921700 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 9607
I20260928 16:42:54.922219 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:41523
--local_ip_for_outbound_sockets=127.25.71.67
--tserver_master_addrs=127.25.71.126:37361
--webserver_port=44519
--webserver_interface=127.25.71.67
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:42:54 dist-test-slave-nqgh krb5kdc[8833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613774, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:54.933756 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:54.916747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:33647 (local address 127.25.71.126:37361)
0928 16:42:54.917056 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:54.917062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:54.918404 (+ 1342us) server_negotiation.cc:407] Connection header received
0928 16:42:54.920017 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:54.920021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:54.920165 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:54.920285 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:54.922084 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:54.922894 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:54.924512 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:54.924912 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:54.925066 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:54.928606 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:54.928611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:54.928613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:54.928659 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:54.930506 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:54.931474 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:54.931478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:54.931479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:54.931637 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:54.932172 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:54.932175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:54.932176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:54.932460 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:54.932573 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:54.933399 (+ 826us) server_negotiation.cc:299] Negotiation successful
0928 16:42:54.933592 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260928 16:42:54.935039 9742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:54.917370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.66:33647)
0928 16:42:54.918175 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:54.918240 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:54.919716 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:54.920527 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:54.920536 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:54.921073 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:54.921812 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:54.921837 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:54.923084 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:54.923087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:54.924329 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:54.924345 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:54.925155 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:54.925159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:54.925351 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:54.926075 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:54.926098 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:54.928215 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:54.930789 (+ 2574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:54.930797 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:54.930822 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:54.931285 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:54.931776 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:54.931779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:54.931782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:54.932017 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:54.932625 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:54.932635 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:54.933050 (+ 415us) client_negotiation.cc:769] Sending connection context
0928 16:42:54.933331 (+ 281us) client_negotiation.cc:240] Negotiation successful
0928 16:42:54.933761 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":198,"threads_started":1}
I20260928 16:42:54.936949 9740 heartbeater.cc:347] Connected to a master server at 127.25.71.126:37361
I20260928 16:42:54.937371 9740 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:54.938078 9740 heartbeater.cc:511] Master 127.25.71.126:37361 requested a full tablet report, sending...
I20260928 16:42:54.939488 9410 ts_manager.cc:194] Registered new tserver with Master: a3c79619df0b47b8a1700d9d207151f5 (127.25.71.66:46519)
I20260928 16:42:54.940371 9410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:33647
W20260928 16:42:55.057833 9743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:42:55.058156 9743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:42:55.058204 9743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:42:55.062889 9743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:42:55.063021 9743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:42:55.063112 9743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:42:55.069355 9743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:41523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=44519
--tserver_master_addrs=127.25.71.126:37361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:42:55.070596 9743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:42:55.071748 9743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:42:55.084415 9748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:42:55.084522 9751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:55.084743 9743 server_base.cc:1034] running on GCE node
W20260928 16:42:55.084416 9749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:42:55.085450 9743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:42:55.086289 9743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:42:55.087600 9743 hybrid_clock.cc:648] HybridClock initialized: now 1790613775087569 us; error 68 us; skew 500 ppm
Sep 28 16:42:55 dist-test-slave-nqgh krb5kdc[8833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613775, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:42:55.091256 9743 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:42:55.092419 9743 webserver.cc:460] Webserver started at http://127.25.71.67:44519/ using document root <none> and password file <none>
I20260928 16:42:55.093079 9743 fs_manager.cc:359] Metadata directory not provided
I20260928 16:42:55.093173 9743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:42:55.098143 9743 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:42:55.101310 9758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:55.103427 9743 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:42:55.103619 9743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "05db9f8c71e84e3e954c351ee1a54f0a"
format_stamp: "Formatted at 2026-09-28 16:42:33 on dist-test-slave-nqgh"
server_key: "6c2dc4e8fb2353e1a489d0e175990ee9"
server_key_iv: "943be50e8343cacd99855807c854de4f"
server_key_version: "encryptionkey@0"
I20260928 16:42:55.104297 9743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:42:55.120402 9743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:42:55.124059 9743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:42:55.124272 9743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:42:55.124950 9743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:42:55.125928 9743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:42:55.125986 9743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:55.126044 9743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:42:55.126075 9743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:42:55.137012 9743 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:41523
I20260928 16:42:55.137080 9871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:41523 every 8 connection(s)
I20260928 16:42:55.138214 9743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:42:55.142467 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 9743
Sep 28 16:42:55 dist-test-slave-nqgh krb5kdc[8833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613775, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:42:55.154774 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:55.143359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54730 (local address 127.25.71.126:37361)
0928 16:42:55.143759 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:55.143765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:55.143783 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:42:55.143838 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:55.143841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:55.143889 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:55.143980 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:55.144810 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.145652 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.146610 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.146936 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.147037 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:55.149230 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:55.149236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:55.149239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:55.149291 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:55.151717 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:55.152716 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:55.152721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:55.152722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:55.152861 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:55.153381 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:55.153385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:55.153387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:55.153693 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:55.154001 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:55.154340 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 16:42:55.154513 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":164,"threads_started":1}
I20260928 16:42:55.160075 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:55.141001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:58431 (local address 127.25.71.126:37361)
0928 16:42:55.141303 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:55.141307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:55.142612 (+ 1305us) server_negotiation.cc:407] Connection header received
0928 16:42:55.144741 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:55.144745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:55.144818 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:55.144902 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:55.146656 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.147258 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.149249 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.149505 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.149602 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:55.154439 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:55.154445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:55.154448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:55.154502 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:55.156449 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:55.157290 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:55.157294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:55.157296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:55.157420 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:55.157966 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:55.157969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:55.157971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:55.158317 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:55.158417 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:55.159702 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0928 16:42:55.159890 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":166}
I20260928 16:42:55.161039 9874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:55.141559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.67:58431)
0928 16:42:55.142372 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:55.142444 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:55.144379 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:55.145197 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:55.145208 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:55.145717 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:42:55.146422 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:55.146442 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.147421 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.147425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:55.148999 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:55.149030 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.149733 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.149738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:55.150039 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:55.151037 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:42:55.151162 (+ 125us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:42:55.154116 (+ 2954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:42:55.156654 (+ 2538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:55.156664 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:55.156683 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:55.157149 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:55.157568 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:42:55.157582 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:42:55.157585 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:42:55.157830 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:42:55.158463 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:42:55.158471 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:42:55.159229 (+ 758us) client_negotiation.cc:769] Sending connection context
0928 16:42:55.159564 (+ 335us) client_negotiation.cc:240] Negotiation successful
0928 16:42:55.159999 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":185,"threads_started":1}
I20260928 16:42:55.162650 9872 heartbeater.cc:347] Connected to a master server at 127.25.71.126:37361
I20260928 16:42:55.163123 9872 heartbeater.cc:464] Registering TS with master...
I20260928 16:42:55.163877 9872 heartbeater.cc:511] Master 127.25.71.126:37361 requested a full tablet report, sending...
I20260928 16:42:55.165411 9410 ts_manager.cc:194] Registered new tserver with Master: 05db9f8c71e84e3e954c351ee1a54f0a (127.25.71.67:41523)
I20260928 16:42:55.166112 9410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:58431
I20260928 16:42:55.170676 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:42:55.176339 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:55.171615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.25.71.126:37361)
0928 16:42:55.171824 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:55.171829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:55.172100 (+ 271us) server_negotiation.cc:407] Connection header received
0928 16:42:55.172287 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:55.172290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:55.172471 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:55.172630 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:42:55.173047 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.173691 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.174826 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.175088 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.175214 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:55.175610 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:42:55.175824 (+ 214us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 16:42:55.176136 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54740: token expired
Metrics: {"server-negotiator.queue_time_us":85}
W20260928 16:42:55.176483 9604 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54740: token expired
W20260928 16:42:55.177114 9340 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.25.71.126:37361)
I20260928 16:42:55.186523 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:55.177661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.25.71.126:37361)
0928 16:42:55.177823 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:55.177828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:55.177868 (+ 40us) server_negotiation.cc:407] Connection header received
0928 16:42:55.177966 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:55.177969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:55.178020 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:55.178087 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:42:55.178959 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.179533 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.180427 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.180663 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.180760 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:55.182430 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:42:55.182434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:42:55.182436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:42:55.182469 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:42:55.184274 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:55.184818 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:55.184822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:55.184825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:55.184915 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:42:55.185277 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:42:55.185280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:42:55.185282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:42:55.185517 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:42:55.185666 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:55.186118 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 16:42:55.186348 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:42:55.190156 9410 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:42:55.193904 9410 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:42:55.205741 9536 tablet_service.cc:1467] Processing CreateTablet for tablet 7f4661b2bf2d452db24c0ab5c7a8f59b (DEFAULT_TABLE table=test-table [id=de58cc2d564c4ed19c02488e59575d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:42:55.206701 9536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f4661b2bf2d452db24c0ab5c7a8f59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:55.210893 9884 tablet_bootstrap.cc:492] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e: Bootstrap starting.
I20260928 16:42:55.211377 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:55.204116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:41523 (local address 127.0.0.1:40546)
0928 16:42:55.204285 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:55.204301 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:55.204401 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:55.205991 (+ 1590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:55.205995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:55.206028 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:55.206176 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:55.206184 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.207836 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.207840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:55.209239 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:55.209256 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.210818 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.210822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:55.210957 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:55.211045 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 16:42:55.211106 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 16:42:55.211222 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":64,"spinlock_wait_cycles":3840}
I20260928 16:42:55.211776 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:55.204487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:46519 (local address 127.0.0.1:41584)
0928 16:42:55.204898 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:55.204911 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:55.205005 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:55.205982 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:55.205985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:55.206009 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:55.206157 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:55.206165 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.207646 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.207650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:55.209405 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:55.209424 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.210893 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.210899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:55.211025 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:55.211062 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 16:42:55.211123 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 16:42:55.211298 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":150,"thread_start_us":174,"threads_started":1}
I20260928 16:42:55.212119 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:55.205140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41584 (local address 127.25.71.66:46519)
0928 16:42:55.205488 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:55.205498 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:55.205524 (+ 26us) server_negotiation.cc:407] Connection header received
0928 16:42:55.205604 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:55.205608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:55.205798 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:55.205969 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:55.206488 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.207524 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.209616 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.210690 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.211459 (+ 769us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:55.211692 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:55.211766 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 16:42:55.211915 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":105,"threads_started":1}
I20260928 16:42:55.212656 9882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:55.204643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40546 (local address 127.25.71.67:41523)
0928 16:42:55.205329 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:55.205344 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:55.205377 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:42:55.205480 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:55.205486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:55.205784 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:55.206363 (+ 579us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:55.206566 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.207692 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.209449 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.210595 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.212087 (+ 1492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:55.212303 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:55.212378 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 16:42:55.212504 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":190,"threads_started":1}
I20260928 16:42:55.213392 9884 tablet_bootstrap.cc:654] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e: Neither blocks nor log segments found. Creating new log.
I20260928 16:42:55.215315 9884 tablet_bootstrap.cc:492] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e: No bootstrap required, opened a new log
I20260928 16:42:55.215440 9884 ts_tablet_manager.cc:1397] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 16:42:55.215911 9884 raft_consensus.cc:348] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:42:55.216032 9884 raft_consensus.cc:374] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:55.216065 9884 raft_consensus.cc:729] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cda6e131080426490efc2ba9323001e, State: Initialized, Role: FOLLOWER
I20260928 16:42:55.216204 9884 consensus_queue.cc:260] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:42:55.215889 9674 tablet_service.cc:1467] Processing CreateTablet for tablet 7f4661b2bf2d452db24c0ab5c7a8f59b (DEFAULT_TABLE table=test-table [id=de58cc2d564c4ed19c02488e59575d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:42:55.216154 9806 tablet_service.cc:1467] Processing CreateTablet for tablet 7f4661b2bf2d452db24c0ab5c7a8f59b (DEFAULT_TABLE table=test-table [id=de58cc2d564c4ed19c02488e59575d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:42:55.216624 9884 ts_tablet_manager.cc:1428] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 16:42:55.217198 9806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f4661b2bf2d452db24c0ab5c7a8f59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:55.217195 9674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f4661b2bf2d452db24c0ab5c7a8f59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:42:55.221719 9886 raft_consensus.cc:482] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:42:55.221988 9886 raft_consensus.cc:504] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:42:55.224830 9886 leader_election.cc:288] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05db9f8c71e84e3e954c351ee1a54f0a (127.25.71.67:41523), a3c79619df0b47b8a1700d9d207151f5 (127.25.71.66:46519)
I20260928 16:42:55.228768 9890 tablet_bootstrap.cc:492] T 7f4661b2bf2d452db24c0ab5c7a8f59b P a3c79619df0b47b8a1700d9d207151f5: Bootstrap starting.
I20260928 16:42:55.228960 9889 tablet_bootstrap.cc:492] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a: Bootstrap starting.
I20260928 16:42:55.230856 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:55.224340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:41523 (local address 127.25.71.65:59667)
0928 16:42:55.225511 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:55.225531 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:55.225640 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:55.226077 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:55.226080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:55.226097 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:55.226180 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:55.226185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.227171 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.227174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:55.228776 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:55.228815 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.230213 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.230218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:55.230338 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:55.230428 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 16:42:55.230514 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 16:42:55.230620 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1106,"mutex_wait_us":137,"thread_start_us":94,"threads_started":1}
I20260928 16:42:55.231431 9882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:55.224745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:59667 (local address 127.25.71.67:41523)
0928 16:42:55.224867 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:55.224872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:55.225718 (+ 846us) server_negotiation.cc:407] Connection header received
0928 16:42:55.225813 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:55.225816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:55.225885 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:55.225957 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:55.226301 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.227060 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.229251 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.229893 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.230039 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:55.230101 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:55.231219 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0928 16:42:55.231317 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 16:42:55.232640 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:55.224013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:46519 (local address 127.25.71.65:43447)
0928 16:42:55.225137 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:55.225153 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:55.225295 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:55.225726 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:55.225729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:55.225746 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:55.225901 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:55.225908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.227073 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.227076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:55.228612 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:55.228628 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.231958 (+ 3330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.231963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:55.232142 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:55.232180 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 16:42:55.232278 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 16:42:55.232406 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"mutex_wait_us":78,"thread_start_us":93,"threads_started":1}
I20260928 16:42:55.232694 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:55.224451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:43447 (local address 127.25.71.66:46519)
0928 16:42:55.224614 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:55.224619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:55.225241 (+ 622us) server_negotiation.cc:407] Connection header received
0928 16:42:55.225440 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:55.225444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:55.225506 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:55.225586 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:55.226034 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.226918 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.228780 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.229370 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.229487 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:55.229567 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:55.232396 (+ 2829us) server_negotiation.cc:299] Negotiation successful
0928 16:42:55.232547 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:42:55.232775 9890 tablet_bootstrap.cc:654] T 7f4661b2bf2d452db24c0ab5c7a8f59b P a3c79619df0b47b8a1700d9d207151f5: Neither blocks nor log segments found. Creating new log.
I20260928 16:42:55.232012 9826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4661b2bf2d452db24c0ab5c7a8f59b" candidate_uuid: "7cda6e131080426490efc2ba9323001e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" is_pre_election: true
I20260928 16:42:55.234088 9890 log.cc:824] T 7f4661b2bf2d452db24c0ab5c7a8f59b P a3c79619df0b47b8a1700d9d207151f5: Log is configured to *not* fsync() on all Append() calls
W20260928 16:42:55.234661 9490 leader_election.cc:341] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05db9f8c71e84e3e954c351ee1a54f0a (127.25.71.67:41523): Illegal state: must be running to vote when last-logged opid is not known
I20260928 16:42:55.234864 9889 tablet_bootstrap.cc:654] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a: Neither blocks nor log segments found. Creating new log.
I20260928 16:42:55.233821 9694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4661b2bf2d452db24c0ab5c7a8f59b" candidate_uuid: "7cda6e131080426490efc2ba9323001e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c79619df0b47b8a1700d9d207151f5" is_pre_election: true
I20260928 16:42:55.235821 9889 log.cc:824] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a: Log is configured to *not* fsync() on all Append() calls
W20260928 16:42:55.236274 9488 leader_election.cc:341] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3c79619df0b47b8a1700d9d207151f5 (127.25.71.66:46519): Illegal state: must be running to vote when last-logged opid is not known
I20260928 16:42:55.236421 9488 leader_election.cc:302] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7cda6e131080426490efc2ba9323001e; no voters: 05db9f8c71e84e3e954c351ee1a54f0a, a3c79619df0b47b8a1700d9d207151f5
I20260928 16:42:55.236887 9886 raft_consensus.cc:2726] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 16:42:55.237144 9890 tablet_bootstrap.cc:492] T 7f4661b2bf2d452db24c0ab5c7a8f59b P a3c79619df0b47b8a1700d9d207151f5: No bootstrap required, opened a new log
I20260928 16:42:55.237428 9890 ts_tablet_manager.cc:1397] T 7f4661b2bf2d452db24c0ab5c7a8f59b P a3c79619df0b47b8a1700d9d207151f5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 16:42:55.239332 9889 tablet_bootstrap.cc:492] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a: No bootstrap required, opened a new log
I20260928 16:42:55.239600 9889 ts_tablet_manager.cc:1397] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20260928 16:42:55.241299 9890 raft_consensus.cc:348] T 7f4661b2bf2d452db24c0ab5c7a8f59b P a3c79619df0b47b8a1700d9d207151f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:42:55.241621 9890 raft_consensus.cc:374] T 7f4661b2bf2d452db24c0ab5c7a8f59b P a3c79619df0b47b8a1700d9d207151f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:55.241688 9890 raft_consensus.cc:729] T 7f4661b2bf2d452db24c0ab5c7a8f59b P a3c79619df0b47b8a1700d9d207151f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c79619df0b47b8a1700d9d207151f5, State: Initialized, Role: FOLLOWER
I20260928 16:42:55.242242 9890 consensus_queue.cc:260] T 7f4661b2bf2d452db24c0ab5c7a8f59b P a3c79619df0b47b8a1700d9d207151f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:42:55.242645 9889 raft_consensus.cc:348] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:42:55.242902 9889 raft_consensus.cc:374] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:42:55.242964 9889 raft_consensus.cc:729] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05db9f8c71e84e3e954c351ee1a54f0a, State: Initialized, Role: FOLLOWER
I20260928 16:42:55.243373 9740 heartbeater.cc:503] Master 127.25.71.126:37361 was elected leader, sending a full tablet report...
I20260928 16:42:55.243436 9889 consensus_queue.cc:260] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:42:55.243562 9890 ts_tablet_manager.cc:1428] T 7f4661b2bf2d452db24c0ab5c7a8f59b P a3c79619df0b47b8a1700d9d207151f5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 16:42:55.244280 9889 ts_tablet_manager.cc:1428] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260928 16:42:55.244751 9872 heartbeater.cc:503] Master 127.25.71.126:37361 was elected leader, sending a full tablet report...
I20260928 16:42:55.263860 9894 raft_consensus.cc:482] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:42:55.264206 9894 raft_consensus.cc:504] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:42:55.265743 9894 leader_election.cc:288] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3c79619df0b47b8a1700d9d207151f5 (127.25.71.66:46519), 7cda6e131080426490efc2ba9323001e (127.25.71.65:35451)
I20260928 16:42:55.271744 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:55.266249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:53601 (local address 127.25.71.66:46519)
0928 16:42:55.266453 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:55.266459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:55.266480 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:42:55.266734 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:55.266739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:55.266806 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:55.266879 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:55.267264 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.268103 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.269958 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.270761 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.270941 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:55.271022 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:55.271428 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 16:42:55.271538 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:42:55.271744 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:55.266313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:56731 (local address 127.25.71.65:35451)
0928 16:42:55.266727 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:55.266732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:55.266773 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:42:55.266964 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:55.266968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:55.267025 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:55.267106 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:42:55.267390 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.268154 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.269958 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.270802 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.270941 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:55.271111 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:55.271428 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 16:42:55.271590 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":165,"threads_started":1}
I20260928 16:42:55.271746 9874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:55.266076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:46519 (local address 127.25.71.67:53601)
0928 16:42:55.266382 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:55.266400 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:55.266520 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:55.266981 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:55.266983 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:55.266999 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:55.267126 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:55.267133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.268260 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.268264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:55.269756 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:55.269771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.271013 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.271017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:55.271213 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:55.271247 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 16:42:55.271364 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 16:42:55.271507 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":220}
I20260928 16:42:55.272221 9895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:55.266200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:35451 (local address 127.25.71.67:56731)
0928 16:42:55.266689 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:55.266704 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:42:55.266828 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:42:55.267147 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:42:55.267150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:42:55.267164 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:42:55.267276 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:55.267281 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.268295 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.268298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:55.269756 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:42:55.269771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.271013 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.271017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:42:55.271202 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:55.271247 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 16:42:55.271353 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 16:42:55.271531 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":153,"thread_start_us":201,"threads_started":1}
I20260928 16:42:55.272670 9694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4661b2bf2d452db24c0ab5c7a8f59b" candidate_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c79619df0b47b8a1700d9d207151f5" is_pre_election: true
I20260928 16:42:55.273025 9694 raft_consensus.cc:2454] T 7f4661b2bf2d452db24c0ab5c7a8f59b P a3c79619df0b47b8a1700d9d207151f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05db9f8c71e84e3e954c351ee1a54f0a in term 0.
I20260928 16:42:55.273120 9554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4661b2bf2d452db24c0ab5c7a8f59b" candidate_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cda6e131080426490efc2ba9323001e" is_pre_election: true
I20260928 16:42:55.273422 9554 raft_consensus.cc:2454] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05db9f8c71e84e3e954c351ee1a54f0a in term 0.
I20260928 16:42:55.273535 9759 leader_election.cc:302] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05db9f8c71e84e3e954c351ee1a54f0a, a3c79619df0b47b8a1700d9d207151f5; no voters:
I20260928 16:42:55.273985 9894 raft_consensus.cc:2781] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:42:55.274217 9894 raft_consensus.cc:482] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:42:55.274291 9894 raft_consensus.cc:3037] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:55.275719 9894 raft_consensus.cc:504] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:42:55.276217 9894 leader_election.cc:288] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a [CANDIDATE]: Term 1 election: Requested vote from peers a3c79619df0b47b8a1700d9d207151f5 (127.25.71.66:46519), 7cda6e131080426490efc2ba9323001e (127.25.71.65:35451)
I20260928 16:42:55.277012 9694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4661b2bf2d452db24c0ab5c7a8f59b" candidate_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c79619df0b47b8a1700d9d207151f5"
I20260928 16:42:55.277190 9694 raft_consensus.cc:3037] T 7f4661b2bf2d452db24c0ab5c7a8f59b P a3c79619df0b47b8a1700d9d207151f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:55.277318 9554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4661b2bf2d452db24c0ab5c7a8f59b" candidate_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cda6e131080426490efc2ba9323001e"
I20260928 16:42:55.277546 9554 raft_consensus.cc:3037] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:42:55.278760 9694 raft_consensus.cc:2454] T 7f4661b2bf2d452db24c0ab5c7a8f59b P a3c79619df0b47b8a1700d9d207151f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05db9f8c71e84e3e954c351ee1a54f0a in term 1.
I20260928 16:42:55.279085 9554 raft_consensus.cc:2454] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05db9f8c71e84e3e954c351ee1a54f0a in term 1.
I20260928 16:42:55.279268 9759 leader_election.cc:302] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05db9f8c71e84e3e954c351ee1a54f0a, a3c79619df0b47b8a1700d9d207151f5; no voters:
I20260928 16:42:55.279524 9894 raft_consensus.cc:2781] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:42:55.279815 9894 raft_consensus.cc:686] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a [term 1 LEADER]: Becoming Leader. State: Replica: 05db9f8c71e84e3e954c351ee1a54f0a, State: Running, Role: LEADER
I20260928 16:42:55.280218 9894 consensus_queue.cc:237] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:42:55.284623 9410 catalog_manager.cc:5495] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 05db9f8c71e84e3e954c351ee1a54f0a (127.25.71.67). New cstate: current_term: 1 leader_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } health_report { overall_health: UNKNOWN } } }
I20260928 16:42:55.300483 9882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:55.296148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40550 (local address 127.25.71.67:41523)
0928 16:42:55.296311 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 16:42:55.296316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:42:55.296338 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:42:55.296505 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:42:55.296508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:42:55.296577 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:42:55.296708 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:42:55.297065 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.297639 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.298701 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:42:55.298977 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:42:55.299082 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:42:55.299468 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:42:55.299982 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:42:55.300094 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:42:55.300238 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 16:42:55.300332 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:42:55.308645 9694 raft_consensus.cc:1261] T 7f4661b2bf2d452db24c0ab5c7a8f59b P a3c79619df0b47b8a1700d9d207151f5 [term 1 FOLLOWER]: Refusing update from remote peer 05db9f8c71e84e3e954c351ee1a54f0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:42:55.308641 9554 raft_consensus.cc:1261] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Refusing update from remote peer 05db9f8c71e84e3e954c351ee1a54f0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:42:55.309561 9897 consensus_queue.cc:1035] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:42:55.309902 9894 consensus_queue.cc:1035] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:42:55.316636 9900 mvcc.cc:205] Tried to move back new op lower bound from 7334354023653773312 to 7334354023551397888. Current Snapshot: MvccSnapshot[applied={T|T < 7334354023653773312}]
I20260928 16:42:55.330518 9414 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:42:55.330763 9414 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:42:55.334352 9414 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee: Sending DeleteTablet for 3 replicas of tablet 7f4661b2bf2d452db24c0ab5c7a8f59b
I20260928 16:42:55.335305 9536 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f4661b2bf2d452db24c0ab5c7a8f59b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:42:55 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:56564
I20260928 16:42:55.335387 9674 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f4661b2bf2d452db24c0ab5c7a8f59b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:42:55 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:41584
I20260928 16:42:55.335822 9806 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f4661b2bf2d452db24c0ab5c7a8f59b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:42:55 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:40546
I20260928 16:42:55.336298 9910 tablet_replica.cc:330] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e: stopping tablet replica
I20260928 16:42:55.336325 9912 tablet_replica.cc:330] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a: stopping tablet replica
I20260928 16:42:55.336465 9910 raft_consensus.cc:2229] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:42:55.336567 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 9452
I20260928 16:42:55.336623 9910 raft_consensus.cc:2258] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:42:55.336663 9912 raft_consensus.cc:2229] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:42:55.337554 9912 raft_consensus.cc:2258] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:42:55.338084 9911 tablet_replica.cc:330] T 7f4661b2bf2d452db24c0ab5c7a8f59b P a3c79619df0b47b8a1700d9d207151f5: stopping tablet replica
I20260928 16:42:55.338454 9911 raft_consensus.cc:2229] T 7f4661b2bf2d452db24c0ab5c7a8f59b P a3c79619df0b47b8a1700d9d207151f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:42:55.338903 9911 raft_consensus.cc:2258] T 7f4661b2bf2d452db24c0ab5c7a8f59b P a3c79619df0b47b8a1700d9d207151f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:42:55.339375 9912 ts_tablet_manager.cc:1905] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:42:55.341706 9911 ts_tablet_manager.cc:1905] T 7f4661b2bf2d452db24c0ab5c7a8f59b P a3c79619df0b47b8a1700d9d207151f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:42:55.342540 9912 ts_tablet_manager.cc:1918] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:42:55.342690 9912 log.cc:1196] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/7f4661b2bf2d452db24c0ab5c7a8f59b
I20260928 16:42:55.343053 9912 ts_tablet_manager.cc:1939] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 05db9f8c71e84e3e954c351ee1a54f0a: Deleting consensus metadata
I20260928 16:42:55.344815 9400 catalog_manager.cc:4841] TS 05db9f8c71e84e3e954c351ee1a54f0a (127.25.71.67:41523): tablet 7f4661b2bf2d452db24c0ab5c7a8f59b (table test-table [id=de58cc2d564c4ed19c02488e59575d7c]) successfully deleted
I20260928 16:42:55.345213 9911 ts_tablet_manager.cc:1918] T 7f4661b2bf2d452db24c0ab5c7a8f59b P a3c79619df0b47b8a1700d9d207151f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:42:55.345329 9911 log.cc:1196] T 7f4661b2bf2d452db24c0ab5c7a8f59b P a3c79619df0b47b8a1700d9d207151f5: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/7f4661b2bf2d452db24c0ab5c7a8f59b
I20260928 16:42:55.345606 9911 ts_tablet_manager.cc:1939] T 7f4661b2bf2d452db24c0ab5c7a8f59b P a3c79619df0b47b8a1700d9d207151f5: Deleting consensus metadata
I20260928 16:42:55.346850 9398 catalog_manager.cc:4841] TS a3c79619df0b47b8a1700d9d207151f5 (127.25.71.66:46519): tablet 7f4661b2bf2d452db24c0ab5c7a8f59b (table test-table [id=de58cc2d564c4ed19c02488e59575d7c]) successfully deleted
W20260928 16:42:55.348156 9399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.71.65:35451 (error 108)
I20260928 16:42:55.349709 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:42:55.349215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:35451 (local address 127.0.0.1:56578)
0928 16:42:55.349399 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 16:42:55.349510 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.65:35451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 16:42:55.349897 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 9607
W20260928 16:42:55.350760 9399 catalog_manager.cc:4584] TS 7cda6e131080426490efc2ba9323001e (127.25.71.65:35451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f4661b2bf2d452db24c0ab5c7a8f59b: Network error: Client connection negotiation failed: client connection to 127.25.71.65:35451: connect: Connection refused (error 111)
I20260928 16:42:55.356555 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 9743
I20260928 16:42:55.362862 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 9381
I20260928 16:43:15.371436 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:37361
--webserver_interface=127.25.71.126
--webserver_port=44169
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:37361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:15.512745 9914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:15.513165 9914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:15.513217 9914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:15.518002 9914 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 16:43:15.518122 9914 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 16:43:15.518138 9914 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 16:43:15.518153 9914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:15.518173 9914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:15.518193 9914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:15.518208 9914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:15.518224 9914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:15.524902 9914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:37361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:37361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=44169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:15.526515 9914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:15.527923 9914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:15.540235 9919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:15.540333 9914 server_base.cc:1034] running on GCE node
W20260928 16:43:15.540223 9920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:15.540185 9922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:15.541340 9914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:15.543107 9914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:15.544342 9914 hybrid_clock.cc:648] HybridClock initialized: now 1790613795544312 us; error 54 us; skew 500 ppm
Sep 28 16:43:15 dist-test-slave-nqgh krb5kdc[8833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613795, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:15.548295 9914 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:43:15.549973 9914 webserver.cc:460] Webserver started at http://127.25.71.126:44169/ using document root <none> and password file <none>
I20260928 16:43:15.550870 9914 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:15.550968 9914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:15.556523 9914 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:43:15.559952 9929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:15.561877 9914 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:43:15.562120 9914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "56436701b863475f99bf93e6d98fadee"
format_stamp: "Formatted at 2026-09-28 16:42:32 on dist-test-slave-nqgh"
server_key: "4d9c43daffb57ecb6f42410e9709b4c5"
server_key_iv: "71e1f4d009986b959b066e02f112f737"
server_key_version: "encryptionkey@0"
I20260928 16:43:15.563066 9914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:15.586555 9914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:15.591068 9914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:15.591311 9914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:15.601416 9914 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:37361
I20260928 16:43:15.601748 9981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:37361 every 8 connection(s)
I20260928 16:43:15.603066 9914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:15.608379 9982 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:43:15.610222 9982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee: Bootstrap starting.
I20260928 16:43:15.611595 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 9914
I20260928 16:43:15.612118 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:35451
--local_ip_for_outbound_sockets=127.25.71.65
--tserver_master_addrs=127.25.71.126:37361
--webserver_port=36501
--webserver_interface=127.25.71.65
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:43:15.628506 9982 log.cc:824] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:15.641624 9982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:43:15.642134 9982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee: Bootstrap complete.
I20260928 16:43:15.646803 9982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56436701b863475f99bf93e6d98fadee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37361 } }
I20260928 16:43:15.647303 9982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56436701b863475f99bf93e6d98fadee, State: Initialized, Role: FOLLOWER
I20260928 16:43:15.647891 9982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56436701b863475f99bf93e6d98fadee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37361 } }
I20260928 16:43:15.648037 9982 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:15.648113 9982 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:15.648211 9982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 2 FOLLOWER]: Advancing to term 3
I20260928 16:43:15.650475 9982 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56436701b863475f99bf93e6d98fadee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37361 } }
I20260928 16:43:15.650902 9982 leader_election.cc:302] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56436701b863475f99bf93e6d98fadee; no voters:
I20260928 16:43:15.651247 9982 leader_election.cc:288] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 16:43:15.651369 9986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 3 FOLLOWER]: Leader election won for term 3
I20260928 16:43:15.651616 9986 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [term 3 LEADER]: Becoming Leader. State: Replica: 56436701b863475f99bf93e6d98fadee, State: Running, Role: LEADER
I20260928 16:43:15.651986 9986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56436701b863475f99bf93e6d98fadee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37361 } }
I20260928 16:43:15.652757 9982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:15.653553 9987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "56436701b863475f99bf93e6d98fadee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56436701b863475f99bf93e6d98fadee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37361 } } }
I20260928 16:43:15.653813 9987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:15.654410 9988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56436701b863475f99bf93e6d98fadee. Latest consensus state: current_term: 3 leader_uuid: "56436701b863475f99bf93e6d98fadee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56436701b863475f99bf93e6d98fadee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37361 } } }
I20260928 16:43:15.654588 9988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:15.655011 9995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:15.662225 9995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:15.663370 9995 catalog_manager.cc:1259] Loaded cluster ID: 76d1f907ef46487dacb3e5e99b1831a8
I20260928 16:43:15.663471 9995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:15.666430 9995 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:15.667546 9995 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee: Loaded TSK: 0
I20260928 16:43:15.667670 9995 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 56436701b863475f99bf93e6d98fadee: Loaded TSK: 1
F20260928 16:43:15.668728 9995 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe8d3ec49ad google::LogMessage::Fail() at ??:0
@ 0x7fe8d3ec788a google::LogMessage::SendToLog() at ??:0
@ 0x7fe8d3ec44de google::LogMessage::Flush() at ??:0
@ 0x7fe8d3ec5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe8d6b8818d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe8d6b80f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe8de82d7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe8de80799c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe8de808178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fe8de849df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe8de86882b std::function<>::operator()() at ??:0
@ 0x7fe8de807b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe8de808f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe8de805579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe8de849640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe8d61163d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe8d6116c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe8d61183f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe8d6107512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe8d774b6db start_thread at ??:0
@ 0x7fe8d29c261f clone at ??:0
W20260928 16:43:15.799401 9984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:15.799983 9984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:15.800067 9984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:15.808084 9984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:15.808249 9984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:15.808364 9984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:43:15.818831 9984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:35451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=36501
--tserver_master_addrs=127.25.71.126:37361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:15.820413 9984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:15.821849 9984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:15.835086 10029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:15.835337 10030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:15.835062 10032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:15.835785 9984 server_base.cc:1034] running on GCE node
I20260928 16:43:15.836421 9984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:15.837441 9984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:15.838701 9984 hybrid_clock.cc:648] HybridClock initialized: now 1790613795838654 us; error 62 us; skew 500 ppm
Sep 28 16:43:15 dist-test-slave-nqgh krb5kdc[8833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613795, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:15.842963 9984 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:43:15.844374 9984 webserver.cc:460] Webserver started at http://127.25.71.65:36501/ using document root <none> and password file <none>
I20260928 16:43:15.845119 9984 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:15.845244 9984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:15.851011 9984 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:43:15.854113 10039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:15.856302 9984 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:43:15.856529 9984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "7cda6e131080426490efc2ba9323001e"
format_stamp: "Formatted at 2026-09-28 16:42:33 on dist-test-slave-nqgh"
server_key: "97fe083de4df83fed0d0b95a2a0ac00c"
server_key_iv: "65ae4e60d3db4c43e0c84473c6997e2f"
server_key_version: "encryptionkey@0"
I20260928 16:43:15.857326 9984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:15.881304 9984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:15.885442 9984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:15.885699 9984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:15.886519 9984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:15.888041 10046 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 16:43:15.893592 9984 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 16:43:15.893711 9984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260928 16:43:15.893783 9984 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 16:43:15.896265 9984 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 16:43:15.896363 9984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260928 16:43:15.896736 10046 tablet_bootstrap.cc:492] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e: Bootstrap starting.
I20260928 16:43:15.909508 9984 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:35451
I20260928 16:43:15.911435 10153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:35451 every 8 connection(s)
I20260928 16:43:15.912715 9984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:43:15.922739 10046 log.cc:824] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:15.922955 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 9984
I20260928 16:43:15.923512 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:46519
--local_ip_for_outbound_sockets=127.25.71.66
--tserver_master_addrs=127.25.71.126:37361
--webserver_port=39663
--webserver_interface=127.25.71.66
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:43:15.932572 10046 tablet_bootstrap.cc:492] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:43:15.933017 10046 tablet_bootstrap.cc:492] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e: Bootstrap complete.
I20260928 16:43:15.934178 10046 ts_tablet_manager.cc:1397] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e: Time spent bootstrapping tablet: real 0.038s user 0.022s sys 0.012s
I20260928 16:43:15.936820 10046 raft_consensus.cc:348] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:43:15.937278 10046 raft_consensus.cc:729] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cda6e131080426490efc2ba9323001e, State: Initialized, Role: FOLLOWER
I20260928 16:43:15.937780 10046 consensus_queue.cc:260] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:43:15.938863 10046 ts_tablet_manager.cc:1428] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
W20260928 16:43:15.940804 10156 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:43:15.916219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.65:43871)
0928 16:43:15.917177 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:15.917269 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:15.918618 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:15.939935 (+ 21317us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":290,"threads_started":1}
W20260928 16:43:15.941605 10154 heartbeater.cc:644] Failed to heartbeat to 127.25.71.126:37361 (0 consecutive failures): Network error: Failed to ping master at 127.25.71.126:37361: Client connection negotiation failed: client connection to 127.25.71.126:37361: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 16:43:15.942574 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:15.942267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.65:51833)
0928 16:43:15.942424 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:15.942499 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 16:43:15.943707 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:15.943443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.65:41047)
0928 16:43:15.943573 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:15.943632 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 16:43:15.944005 10154 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:43:16.069509 10158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:16.069967 10158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:16.070066 10158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:16.075383 10158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:16.075497 10158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:16.075609 10158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:43:16.082444 10158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:46519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=39663
--tserver_master_addrs=127.25.71.126:37361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:16.084096 10158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:16.085487 10158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:16.098511 10167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:16.098683 10165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:16.098765 10158 server_base.cc:1034] running on GCE node
W20260928 16:43:16.098683 10164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:16.099488 10158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:16.100241 10158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:16.101457 10158 hybrid_clock.cc:648] HybridClock initialized: now 1790613796101426 us; error 48 us; skew 500 ppm
Sep 28 16:43:16 dist-test-slave-nqgh krb5kdc[8833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613796, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:16.108683 10158 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:43:16.109887 10158 webserver.cc:460] Webserver started at http://127.25.71.66:39663/ using document root <none> and password file <none>
I20260928 16:43:16.110567 10158 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:16.110688 10158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:16.115679 10158 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:43:16.118953 10174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:16.120924 10158 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:43:16.121135 10158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "a3c79619df0b47b8a1700d9d207151f5"
format_stamp: "Formatted at 2026-09-28 16:42:33 on dist-test-slave-nqgh"
server_key: "8b6d9062e3534af17230afd9557bde8d"
server_key_iv: "f82ccfa990fce72a726c4a62920969d2"
server_key_version: "encryptionkey@0"
I20260928 16:43:16.121834 10158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:16.148681 10158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:16.152251 10158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:16.152462 10158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:16.153146 10158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:16.154160 10158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:16.154214 10158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:16.154273 10158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:16.154300 10158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:16.165135 10158 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:46519
I20260928 16:43:16.165166 10287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:46519 every 8 connection(s)
I20260928 16:43:16.166334 10158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:43:16.167882 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 10158
I20260928 16:43:16.168395 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:41523
--local_ip_for_outbound_sockets=127.25.71.67
--tserver_master_addrs=127.25.71.126:37361
--webserver_port=44519
--webserver_interface=127.25.71.67
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:43:16.171937 10291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:16.169397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.66:60705)
0928 16:43:16.170067 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:16.170245 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":189,"threads_started":1}
W20260928 16:43:16.172866 10288 heartbeater.cc:644] Failed to heartbeat to 127.25.71.126:37361 (0 consecutive failures): Network error: Failed to ping master at 127.25.71.126:37361: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
I20260928 16:43:16.173907 10291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:16.173565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.66:46169)
0928 16:43:16.173754 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:16.173818 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 16:43:16.174938 10291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:16.174679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.66:51963)
0928 16:43:16.174817 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:16.174870 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 16:43:16.175155 10288 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:43:16.295153 10290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:16.295454 10290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:16.295500 10290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:16.299751 10290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:16.299841 10290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:16.299906 10290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:43:16.305627 10290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:41523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=44519
--tserver_master_addrs=127.25.71.126:37361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:16.307075 10290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:16.308154 10290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:16.319156 10299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:16.319451 10296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:16.319656 10290 server_base.cc:1034] running on GCE node
W20260928 16:43:16.319451 10297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:16.320179 10290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:16.320909 10290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:16.322139 10290 hybrid_clock.cc:648] HybridClock initialized: now 1790613796322103 us; error 53 us; skew 500 ppm
Sep 28 16:43:16 dist-test-slave-nqgh krb5kdc[8833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613796, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:16.325835 10290 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:43:16.327092 10290 webserver.cc:460] Webserver started at http://127.25.71.67:44519/ using document root <none> and password file <none>
I20260928 16:43:16.327740 10290 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:16.327819 10290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:16.332336 10290 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 16:43:16.335572 10306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:16.337541 10290 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:43:16.337740 10290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "05db9f8c71e84e3e954c351ee1a54f0a"
format_stamp: "Formatted at 2026-09-28 16:42:33 on dist-test-slave-nqgh"
server_key: "6c2dc4e8fb2353e1a489d0e175990ee9"
server_key_iv: "943be50e8343cacd99855807c854de4f"
server_key_version: "encryptionkey@0"
I20260928 16:43:16.338406 10290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:16.349907 10290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:16.353407 10290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:16.353605 10290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:16.354231 10290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:16.355201 10290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:16.355254 10290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:16.355319 10290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:16.355343 10290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:16.365769 10290 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:41523
I20260928 16:43:16.365864 10419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:41523 every 8 connection(s)
I20260928 16:43:16.367051 10290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:43:16.372547 10422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:16.370177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.67:57465)
0928 16:43:16.371020 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:16.371193 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":219,"threads_started":1}
W20260928 16:43:16.373328 10420 heartbeater.cc:644] Failed to heartbeat to 127.25.71.126:37361 (0 consecutive failures): Network error: Failed to ping master at 127.25.71.126:37361: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
I20260928 16:43:16.374261 10422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:16.373938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.67:43123)
0928 16:43:16.374111 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:16.374188 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 16:43:16.375182 10422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:16.374968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.67:42659)
0928 16:43:16.375078 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:16.375128 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 16:43:16.375375 10420 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 16:43:16.376320 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 10290
I20260928 16:43:16.945556 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:16.944896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.65:34735)
0928 16:43:16.945307 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:16.945415 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":186,"threads_started":1}
I20260928 16:43:17.176657 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:17.176085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.66:56417)
0928 16:43:17.176429 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:17.176501 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":94,"threads_started":1}
I20260928 16:43:17.377079 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:17.376355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.67:48209)
0928 16:43:17.376759 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:17.376929 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":108,"threads_started":1}
I20260928 16:43:17.514477 10427 raft_consensus.cc:482] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:17.514843 10427 raft_consensus.cc:504] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:43:17.516359 10427 leader_election.cc:288] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05db9f8c71e84e3e954c351ee1a54f0a (127.25.71.67:41523), a3c79619df0b47b8a1700d9d207151f5 (127.25.71.66:46519)
Sep 28 16:43:17 dist-test-slave-nqgh krb5kdc[8833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613795, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.66@KRBTEST.COM
Sep 28 16:43:17 dist-test-slave-nqgh krb5kdc[8833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613795, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.67@KRBTEST.COM
I20260928 16:43:17.533440 10431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:17.517388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:43057 (local address 127.25.71.67:41523)
0928 16:43:17.517782 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:17.517807 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:17.517851 (+ 44us) server_negotiation.cc:407] Connection header received
0928 16:43:17.518702 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:17.518709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:17.519179 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:17.519520 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:17.521017 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:17.522118 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:17.523977 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:17.524539 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:17.524697 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:17.528156 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:17.528162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:17.528178 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:17.528215 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:17.530245 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:17.531103 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:17.531111 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:17.531116 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:17.531269 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:17.531650 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:17.531654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:17.531656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:17.532154 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:17.532285 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:17.532938 (+ 653us) server_negotiation.cc:299] Negotiation successful
0928 16:43:17.533237 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":107,"threads_started":1}
I20260928 16:43:17.533432 10428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:17.516692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:41523 (local address 127.25.71.65:43057)
0928 16:43:17.517043 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:17.517060 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:17.517184 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:17.519759 (+ 2575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:17.519762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:17.520000 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:17.520741 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:17.520766 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:17.522386 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:17.522392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:17.523782 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:17.523800 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:17.524703 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:17.524708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:17.524909 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:17.525747 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:17.525762 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:17.527870 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:17.530408 (+ 2538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:17.530416 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:17.530432 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:17.530932 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:17.531377 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:17.531380 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:17.531381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:17.531543 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:17.532276 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:17.532283 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:17.532577 (+ 294us) client_negotiation.cc:769] Sending connection context
0928 16:43:17.532775 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 16:43:17.533220 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":110,"spinlock_wait_cycles":24064,"thread_start_us":128,"threads_started":1}
I20260928 16:43:17.533512 10429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:17.516803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:46519 (local address 127.25.71.65:56343)
0928 16:43:17.517251 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:17.517269 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:17.517416 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:17.519380 (+ 1964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:17.519387 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:17.520000 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:17.520741 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:17.520766 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:17.522724 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:17.522729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:17.523773 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:17.523787 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:17.524764 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:17.524768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:17.524911 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:17.525605 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:17.525632 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:17.527617 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:17.530509 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:17.530513 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:17.530515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:17.530974 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:17.531401 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:17.531404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:17.531406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:17.531570 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:17.532378 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:17.532381 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:17.532577 (+ 196us) client_negotiation.cc:769] Sending connection context
0928 16:43:17.532776 (+ 199us) client_negotiation.cc:240] Negotiation successful
0928 16:43:17.533309 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":276,"spinlock_wait_cycles":25984,"thread_start_us":86,"threads_started":1}
I20260928 16:43:17.533869 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:17.517012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:56343 (local address 127.25.71.66:46519)
0928 16:43:17.517610 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:17.517641 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:17.517695 (+ 54us) server_negotiation.cc:407] Connection header received
0928 16:43:17.518447 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:17.518464 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:17.518903 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:17.519229 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:17.521039 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:17.522462 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:17.523953 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:17.524585 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:17.524796 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:17.527998 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:17.528009 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:17.528038 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:17.528088 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:17.530302 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:17.531129 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:17.531136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:17.531143 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:17.531294 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:17.531670 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:17.531672 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:17.531674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:17.532261 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:17.532404 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:17.532936 (+ 532us) server_negotiation.cc:299] Negotiation successful
0928 16:43:17.533282 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":165,"threads_started":1}
W20260928 16:43:17.536785 10040 leader_election.cc:341] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3c79619df0b47b8a1700d9d207151f5 (127.25.71.66:46519): Not found: Tablet not found: 7f4661b2bf2d452db24c0ab5c7a8f59b
W20260928 16:43:17.537225 10042 leader_election.cc:341] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05db9f8c71e84e3e954c351ee1a54f0a (127.25.71.67:41523): Not found: Tablet not found: 7f4661b2bf2d452db24c0ab5c7a8f59b
I20260928 16:43:17.537323 10042 leader_election.cc:302] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7cda6e131080426490efc2ba9323001e; no voters: 05db9f8c71e84e3e954c351ee1a54f0a, a3c79619df0b47b8a1700d9d207151f5
I20260928 16:43:17.537700 10427 raft_consensus.cc:2726] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:43:17.947029 10429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:17.946577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.65:57235)
0928 16:43:17.946835 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:17.946927 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260928 16:43:18.178510 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:18.177760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.66:49407)
0928 16:43:18.178257 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:18.178364 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":126,"threads_started":1}
I20260928 16:43:18.378866 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:18.378073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.67:59551)
0928 16:43:18.378524 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:18.378710 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":84,"threads_started":1}
I20260928 16:43:18.948417 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:18.947991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.65:45547)
0928 16:43:18.948254 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:18.948318 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
I20260928 16:43:19.180239 10435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:19.179538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.66:57557)
0928 16:43:19.179983 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:19.180080 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":103,"threads_started":1}
I20260928 16:43:19.380795 10436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:19.380146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.67:57669)
0928 16:43:19.380567 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:19.380658 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":102,"threads_started":1}
I20260928 16:43:19.381508 10437 raft_consensus.cc:482] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:19.381665 10437 raft_consensus.cc:504] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:43:19.382182 10437 leader_election.cc:288] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05db9f8c71e84e3e954c351ee1a54f0a (127.25.71.67:41523), a3c79619df0b47b8a1700d9d207151f5 (127.25.71.66:46519)
W20260928 16:43:19.383332 10042 leader_election.cc:341] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05db9f8c71e84e3e954c351ee1a54f0a (127.25.71.67:41523): Not found: Tablet not found: 7f4661b2bf2d452db24c0ab5c7a8f59b
W20260928 16:43:19.383463 10040 leader_election.cc:341] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3c79619df0b47b8a1700d9d207151f5 (127.25.71.66:46519): Not found: Tablet not found: 7f4661b2bf2d452db24c0ab5c7a8f59b
I20260928 16:43:19.383611 10042 leader_election.cc:302] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7cda6e131080426490efc2ba9323001e; no voters: 05db9f8c71e84e3e954c351ee1a54f0a, a3c79619df0b47b8a1700d9d207151f5
I20260928 16:43:19.383805 10437 raft_consensus.cc:2726] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:43:19.949916 10438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:19.949294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.65:47885)
0928 16:43:19.949683 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:19.949776 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":86,"threads_started":1}
I20260928 16:43:20.181912 10439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:20.181393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.66:53345)
0928 16:43:20.181708 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:20.181783 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":90,"threads_started":1}
I20260928 16:43:20.382750 10440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:20.382047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.67:47979)
0928 16:43:20.382498 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:20.382587 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":112,"threads_started":1}
I20260928 16:43:20.952106 10441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:20.951375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.65:52593)
0928 16:43:20.951873 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:20.951976 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":104,"threads_started":1}
I20260928 16:43:21.183785 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:21.183018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.66:38099)
0928 16:43:21.183459 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:21.183576 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":90,"threads_started":1}
I20260928 16:43:21.384605 10443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:21.383789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.67:45375)
0928 16:43:21.384344 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:21.384456 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":94,"threads_started":1}
I20260928 16:43:21.953703 10444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:21.953066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.65:45345)
0928 16:43:21.953464 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:21.953559 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":86,"threads_started":1}
I20260928 16:43:22.185601 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:22.184974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.66:44777)
0928 16:43:22.185376 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:22.185478 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":110,"threads_started":1}
I20260928 16:43:22.309978 10446 raft_consensus.cc:482] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:22.310170 10446 raft_consensus.cc:504] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:43:22.310777 10446 leader_election.cc:288] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05db9f8c71e84e3e954c351ee1a54f0a (127.25.71.67:41523), a3c79619df0b47b8a1700d9d207151f5 (127.25.71.66:46519)
W20260928 16:43:22.311702 10042 leader_election.cc:341] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05db9f8c71e84e3e954c351ee1a54f0a (127.25.71.67:41523): Not found: Tablet not found: 7f4661b2bf2d452db24c0ab5c7a8f59b
W20260928 16:43:22.311866 10040 leader_election.cc:341] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3c79619df0b47b8a1700d9d207151f5 (127.25.71.66:46519): Not found: Tablet not found: 7f4661b2bf2d452db24c0ab5c7a8f59b
I20260928 16:43:22.311975 10040 leader_election.cc:302] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7cda6e131080426490efc2ba9323001e; no voters: 05db9f8c71e84e3e954c351ee1a54f0a, a3c79619df0b47b8a1700d9d207151f5
I20260928 16:43:22.312157 10446 raft_consensus.cc:2726] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:43:22.386418 10447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:22.385773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.67:47775)
0928 16:43:22.386200 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:22.386303 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":115,"threads_started":1}
I20260928 16:43:22.955560 10448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:22.954983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.65:60207)
0928 16:43:22.955359 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:22.955446 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
I20260928 16:43:23.187718 10449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.186950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.66:47645)
0928 16:43:23.187456 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:23.187557 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":117,"threads_started":1}
I20260928 16:43:23.388417 10450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.387720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.67:50181)
0928 16:43:23.388178 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:23.388277 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":135,"threads_started":1}
I20260928 16:43:23.957262 10451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.956631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.65:53659)
0928 16:43:23.957042 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:23.957148 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":119,"threads_started":1}
I20260928 16:43:24.133411 10452 raft_consensus.cc:482] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:24.133646 10452 raft_consensus.cc:504] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:43:24.134202 10452 leader_election.cc:288] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05db9f8c71e84e3e954c351ee1a54f0a (127.25.71.67:41523), a3c79619df0b47b8a1700d9d207151f5 (127.25.71.66:46519)
W20260928 16:43:24.135208 10042 leader_election.cc:341] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05db9f8c71e84e3e954c351ee1a54f0a (127.25.71.67:41523): Not found: Tablet not found: 7f4661b2bf2d452db24c0ab5c7a8f59b
W20260928 16:43:24.135422 10040 leader_election.cc:341] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3c79619df0b47b8a1700d9d207151f5 (127.25.71.66:46519): Not found: Tablet not found: 7f4661b2bf2d452db24c0ab5c7a8f59b
I20260928 16:43:24.135550 10040 leader_election.cc:302] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7cda6e131080426490efc2ba9323001e; no voters: 05db9f8c71e84e3e954c351ee1a54f0a, a3c79619df0b47b8a1700d9d207151f5
I20260928 16:43:24.135740 10452 raft_consensus.cc:2726] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:43:24.189741 10453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:24.189028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.66:50983)
0928 16:43:24.189486 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:24.189588 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":144,"threads_started":1}
I20260928 16:43:24.390398 10454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:24.389698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.67:58437)
0928 16:43:24.390076 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:24.390247 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":91,"threads_started":1}
I20260928 16:43:24.959085 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:24.958437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.65:36745)
0928 16:43:24.958879 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:24.958975 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":130,"threads_started":1}
I20260928 16:43:25.191697 10456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:25.190959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.66:59361)
0928 16:43:25.191377 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:25.191473 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":93,"threads_started":1}
I20260928 16:43:25.392297 10457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:25.391579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.67:51413)
0928 16:43:25.392081 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:25.392168 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":110,"threads_started":1}
I20260928 16:43:25.961103 10458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:25.960426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.65:47247)
0928 16:43:25.960902 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:25.960994 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":113,"threads_started":1}
I20260928 16:43:26.193548 10459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:26.192989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.66:46865)
0928 16:43:26.193333 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:26.193406 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":96,"threads_started":1}
I20260928 16:43:26.394172 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:26.393382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.67:38437)
0928 16:43:26.393883 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:26.394003 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":101,"threads_started":1}
I20260928 16:43:26.962966 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:26.962307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.65:58055)
0928 16:43:26.962691 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:26.962796 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":111,"threads_started":1}
I20260928 16:43:27.195609 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:27.194924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.66:35851)
0928 16:43:27.195386 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:27.195483 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":136,"threads_started":1}
I20260928 16:43:27.395960 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:27.395246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.67:58919)
0928 16:43:27.395714 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:27.395820 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":109,"threads_started":1}
I20260928 16:43:27.964457 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:27.963916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.65:57319)
0928 16:43:27.964269 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:27.964345 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":96,"threads_started":1}
I20260928 16:43:28.197319 10465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.196587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.66:47827)
0928 16:43:28.197067 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:28.197169 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":93,"threads_started":1}
I20260928 16:43:28.397796 10466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.397142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.67:44527)
0928 16:43:28.397540 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:28.397658 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":94,"threads_started":1}
I20260928 16:43:28.701167 10467 raft_consensus.cc:482] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:28.701336 10467 raft_consensus.cc:504] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05db9f8c71e84e3e954c351ee1a54f0a" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 41523 } } peers { permanent_uuid: "a3c79619df0b47b8a1700d9d207151f5" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 46519 } } peers { permanent_uuid: "7cda6e131080426490efc2ba9323001e" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 35451 } }
I20260928 16:43:28.701835 10467 leader_election.cc:288] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05db9f8c71e84e3e954c351ee1a54f0a (127.25.71.67:41523), a3c79619df0b47b8a1700d9d207151f5 (127.25.71.66:46519)
W20260928 16:43:28.702824 10042 leader_election.cc:341] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05db9f8c71e84e3e954c351ee1a54f0a (127.25.71.67:41523): Not found: Tablet not found: 7f4661b2bf2d452db24c0ab5c7a8f59b
W20260928 16:43:28.703068 10040 leader_election.cc:341] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3c79619df0b47b8a1700d9d207151f5 (127.25.71.66:46519): Not found: Tablet not found: 7f4661b2bf2d452db24c0ab5c7a8f59b
I20260928 16:43:28.703150 10040 leader_election.cc:302] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7cda6e131080426490efc2ba9323001e; no voters: 05db9f8c71e84e3e954c351ee1a54f0a, a3c79619df0b47b8a1700d9d207151f5
I20260928 16:43:28.703338 10467 raft_consensus.cc:2726] T 7f4661b2bf2d452db24c0ab5c7a8f59b P 7cda6e131080426490efc2ba9323001e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:43:28.966069 10468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.965485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.65:54285)
0928 16:43:28.965843 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:28.965933 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":90,"threads_started":1}
I20260928 16:43:29.199422 10469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:29.198749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.66:53379)
0928 16:43:29.199199 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:29.199289 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":105,"threads_started":1}
I20260928 16:43:29.399529 10470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:29.398948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.67:49989)
0928 16:43:29.399337 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:29.399411 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":97,"threads_started":1}
I20260928 16:43:29.969305 10471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:29.968106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.65:44955)
0928 16:43:29.968897 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:29.969048 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":211,"threads_started":1}
I20260928 16:43:30.201259 10472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:30.200689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.66:36889)
0928 16:43:30.201040 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:30.201114 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
I20260928 16:43:30.401288 10473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:30.400696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.67:48597)
0928 16:43:30.401082 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:30.401170 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":112,"threads_started":1}
I20260928 16:43:30.971663 10474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:30.971022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.65:42931)
0928 16:43:30.971439 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:30.971530 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":84,"threads_started":1}
I20260928 16:43:31.203037 10475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:31.202408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37361 (local address 127.25.71.66:34667)
0928 16:43:31.202825 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:31.202907 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.126:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":108,"threads_started":1}
I20260928 16:43:31.383860 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 9984
I20260928 16:43:31.390735 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 10158
I20260928 16:43:31.398113 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 10290
2026-09-28T16:43:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59780 ms)
[----------] 30 tests from SecurityITest (265719 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:31 dist-test-slave-nqgh krb5kdc[10478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:31 dist-test-slave-nqgh krb5kdc[10478](info): set up 2 sockets
Sep 28 16:43:31 dist-test-slave-nqgh krb5kdc[10478](info): commencing operation
krb5kdc: starting...
W20260928 16:43:32.372619 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:43:32 dist-test-slave-nqgh krb5kdc[10478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:43:32.511463 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:41743
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:35295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:41743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:32.650717 10491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:32.651172 10491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:32.651244 10491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:32.656056 10491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:32.656174 10491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:32.656203 10491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:32.656227 10491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:32.656248 10491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:32.663081 10491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:35295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:41743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:41743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:32.664747 10491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:32.666301 10491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:32.677973 10496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:32.677883 10497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:32.678045 10499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:32.678083 10491 server_base.cc:1034] running on GCE node
I20260928 16:43:32.679152 10491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:32.680940 10491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:32.682188 10491 hybrid_clock.cc:648] HybridClock initialized: now 1790613812682167 us; error 53 us; skew 500 ppm
Sep 28 16:43:32 dist-test-slave-nqgh krb5kdc[10478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613812, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:32.686316 10491 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:43:32.687516 10491 webserver.cc:460] Webserver started at http://127.25.71.126:37213/ using document root <none> and password file <none>
I20260928 16:43:32.688180 10491 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:32.688254 10491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:32.688488 10491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:32.691020 10491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "c7f62471597e46f689d17928a7f5eda1"
format_stamp: "Formatted at 2026-09-28 16:43:32 on dist-test-slave-nqgh"
server_key: "2dca1b4b071b854ef7634743fa12713a"
server_key_iv: "6b91400f9e8ab83df73ed5496b11a39f"
server_key_version: "encryptionkey@0"
I20260928 16:43:32.691648 10491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "c7f62471597e46f689d17928a7f5eda1"
format_stamp: "Formatted at 2026-09-28 16:43:32 on dist-test-slave-nqgh"
server_key: "2dca1b4b071b854ef7634743fa12713a"
server_key_iv: "6b91400f9e8ab83df73ed5496b11a39f"
server_key_version: "encryptionkey@0"
I20260928 16:43:32.696344 10491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:43:32.699827 10506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:32.702286 10491 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260928 16:43:32.702543 10491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "c7f62471597e46f689d17928a7f5eda1"
format_stamp: "Formatted at 2026-09-28 16:43:32 on dist-test-slave-nqgh"
server_key: "2dca1b4b071b854ef7634743fa12713a"
server_key_iv: "6b91400f9e8ab83df73ed5496b11a39f"
server_key_version: "encryptionkey@0"
I20260928 16:43:32.702800 10491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:32.715379 10491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:32.718963 10491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:32.719141 10491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:32.727281 10491 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:41743
I20260928 16:43:32.727344 10558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:41743 every 8 connection(s)
I20260928 16:43:32.728428 10491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:32.730437 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 10491
I20260928 16:43:32.730571 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:43:32.730983 25885 external_mini_cluster.cc:1428] Setting key 07e031612d31af64dd496d69d0385b10
I20260928 16:43:32.732764 10559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:32.741151 10559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1: Bootstrap starting.
Sep 28 16:43:32 dist-test-slave-nqgh krb5kdc[10478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:43:32.744443 10559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:32.745393 10559 log.cc:824] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:32.748356 10559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1: No bootstrap required, opened a new log
I20260928 16:43:32.748512 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:32.732790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55270 (local address 127.25.71.126:41743)
0928 16:43:32.733309 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:32.733326 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:32.733368 (+ 42us) server_negotiation.cc:407] Connection header received
0928 16:43:32.734111 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:32.734132 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:32.734518 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:32.734920 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:32.736031 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:32.737108 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:32.738419 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:32.739166 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:32.739319 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:32.742786 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:32.742793 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:32.742812 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:32.742853 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:32.744678 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:32.745194 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:32.745200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:32.745205 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:32.745343 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:32.745683 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:32.745688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:32.745690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:32.746345 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:32.746615 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:32.746984 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 16:43:32.747385 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":171,"threads_started":1}
I20260928 16:43:32.753036 10559 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f62471597e46f689d17928a7f5eda1" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41743 } }
I20260928 16:43:32.753345 10559 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:32.753382 10559 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7f62471597e46f689d17928a7f5eda1, State: Initialized, Role: FOLLOWER
I20260928 16:43:32.753866 10559 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f62471597e46f689d17928a7f5eda1" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41743 } }
I20260928 16:43:32.754019 10559 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:32.754068 10559 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:32.754179 10559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:32.755478 10559 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f62471597e46f689d17928a7f5eda1" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41743 } }
I20260928 16:43:32.755899 10559 leader_election.cc:302] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7f62471597e46f689d17928a7f5eda1; no voters:
I20260928 16:43:32.756264 10559 leader_election.cc:288] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:32.756528 10564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:32.756906 10564 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1 [term 1 LEADER]: Becoming Leader. State: Replica: c7f62471597e46f689d17928a7f5eda1, State: Running, Role: LEADER
I20260928 16:43:32.757318 10564 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f62471597e46f689d17928a7f5eda1" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41743 } }
I20260928 16:43:32.757871 10559 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:32.759382 10566 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7f62471597e46f689d17928a7f5eda1. Latest consensus state: current_term: 1 leader_uuid: "c7f62471597e46f689d17928a7f5eda1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f62471597e46f689d17928a7f5eda1" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41743 } } }
I20260928 16:43:32.759549 10566 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:32.759862 10565 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7f62471597e46f689d17928a7f5eda1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f62471597e46f689d17928a7f5eda1" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 41743 } } }
I20260928 16:43:32.759997 10565 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1 [sys.catalog]: This master's current role is: LEADER
W20260928 16:43:32.764079 10579 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:43:32.764237 10579 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:43:32.764319 10574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:32.765197 10574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:32.771081 10574 catalog_manager.cc:1347] Generated new cluster ID: 7662775e891340c983680ffdcf2c78f2
I20260928 16:43:32.771220 10574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:32.791199 10574 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:32.792531 10574 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:32.803381 10574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1: Generated new TSK 0
I20260928 16:43:32.804205 10574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
I20260928 16:43:32.849989 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41743
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:35295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:32.975176 10585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:32.975492 10585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:32.975551 10585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:32.979871 10585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:32.979962 10585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:32.980052 10585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:43:32.985829 10585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:35295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:32.987118 10585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:32.988240 10585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:32.999516 10591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:32.999541 10590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:32.999531 10593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:32.999768 10585 server_base.cc:1034] running on GCE node
I20260928 16:43:33.000463 10585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:33.001374 10585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:33.002660 10585 hybrid_clock.cc:648] HybridClock initialized: now 1790613813002590 us; error 52 us; skew 500 ppm
Sep 28 16:43:33 dist-test-slave-nqgh krb5kdc[10478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613813, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:33.006102 10585 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:43:33.007266 10585 webserver.cc:460] Webserver started at http://127.25.71.65:41853/ using document root <none> and password file <none>
I20260928 16:43:33.007929 10585 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:33.008019 10585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:33.008244 10585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:33.010488 10585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "475ee1007bfc477284027e09385adace"
format_stamp: "Formatted at 2026-09-28 16:43:33 on dist-test-slave-nqgh"
server_key: "2655cb2b511f6e071b60d33547d024aa"
server_key_iv: "83b33ed25ba4f5175c90b57b41b467fd"
server_key_version: "encryptionkey@0"
I20260928 16:43:33.011158 10585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "475ee1007bfc477284027e09385adace"
format_stamp: "Formatted at 2026-09-28 16:43:33 on dist-test-slave-nqgh"
server_key: "2655cb2b511f6e071b60d33547d024aa"
server_key_iv: "83b33ed25ba4f5175c90b57b41b467fd"
server_key_version: "encryptionkey@0"
I20260928 16:43:33.015813 10585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:43:33.019997 10600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:33.022310 10585 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:43:33.022516 10585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "475ee1007bfc477284027e09385adace"
format_stamp: "Formatted at 2026-09-28 16:43:33 on dist-test-slave-nqgh"
server_key: "2655cb2b511f6e071b60d33547d024aa"
server_key_iv: "83b33ed25ba4f5175c90b57b41b467fd"
server_key_version: "encryptionkey@0"
I20260928 16:43:33.022742 10585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:33.037873 10585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:33.041528 10585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:33.041736 10585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:33.042411 10585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:33.043418 10585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:33.043471 10585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:33.043519 10585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:33.043550 10585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:33.054499 10585 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:34379
I20260928 16:43:33.054569 10713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:34379 every 8 connection(s)
I20260928 16:43:33.055581 10585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:43:33.059540 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 10585
I20260928 16:43:33.059697 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:43:33.060092 25885 external_mini_cluster.cc:1428] Setting key 0c7fe1017b35442d314af91f6dfa0e80
Sep 28 16:43:33 dist-test-slave-nqgh krb5kdc[10478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613813, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.71.66@KRBTEST.COM; defaulting to no policy
I20260928 16:43:33.074132 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.058155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:44483 (local address 127.25.71.126:41743)
0928 16:43:33.058382 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:33.058388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:33.060003 (+ 1615us) server_negotiation.cc:407] Connection header received
0928 16:43:33.061022 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:33.061026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:33.061098 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:33.061188 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:33.062669 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.063496 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.064949 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.065356 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.065483 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.069040 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:33.069046 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:33.069049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:33.069096 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:33.071122 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:33.072017 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:33.072021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:33.072023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:33.072122 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:33.072618 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:33.072621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:33.072622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:33.072858 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:33.072959 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:33.073815 (+ 856us) server_negotiation.cc:299] Negotiation successful
0928 16:43:33.073986 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.66@KRBTEST.COM" created.
I20260928 16:43:33.075452 10716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.058600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:41743 (local address 127.25.71.65:44483)
0928 16:43:33.059757 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:33.059821 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:33.060792 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:33.061411 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:33.061421 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:33.061915 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:33.062436 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:33.062451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.063698 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.063701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:33.064762 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:33.064775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.065559 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.065563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:33.065741 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.066570 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:33.066590 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:33.068679 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:33.071357 (+ 2678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:33.071367 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:33.071391 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:33.071867 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:33.072240 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:33.072243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:33.072244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:33.072476 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:33.072979 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:33.072986 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:33.073404 (+ 418us) client_negotiation.cc:769] Sending connection context
0928 16:43:33.073718 (+ 314us) client_negotiation.cc:240] Negotiation successful
0928 16:43:33.074151 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"thread_start_us":198,"threads_started":1}
I20260928 16:43:33.077517 10714 heartbeater.cc:347] Connected to a master server at 127.25.71.126:41743
I20260928 16:43:33.077958 10714 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:33.078877 10714 heartbeater.cc:511] Master 127.25.71.126:41743 requested a full tablet report, sending...
I20260928 16:43:33.081456 10518 ts_manager.cc:194] Registered new tserver with Master: 475ee1007bfc477284027e09385adace (127.25.71.65:34379)
I20260928 16:43:33.082854 10518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:44483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
I20260928 16:43:33.100713 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41743
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:35295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:33.241133 10719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:33.241539 10719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:33.241592 10719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:33.246436 10719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:33.246556 10719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:33.246660 10719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:43:33.253330 10719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:35295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:33.254933 10719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:33.256328 10719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:33.269155 10724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:33.269174 10725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:33.269448 10719 server_base.cc:1034] running on GCE node
W20260928 16:43:33.269152 10727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:33.270192 10719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:33.271181 10719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:33.272436 10719 hybrid_clock.cc:648] HybridClock initialized: now 1790613813272397 us; error 66 us; skew 500 ppm
Sep 28 16:43:33 dist-test-slave-nqgh krb5kdc[10478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613813, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:33.276404 10719 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:43:33.277689 10719 webserver.cc:460] Webserver started at http://127.25.71.66:36411/ using document root <none> and password file <none>
I20260928 16:43:33.278342 10719 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:33.278416 10719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:33.278610 10719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:33.280992 10719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "69ff8cfe1eee4f909cc0157a1eb0b44a"
format_stamp: "Formatted at 2026-09-28 16:43:33 on dist-test-slave-nqgh"
server_key: "73b92ddaa03aacd9e8b1e4d1b94794d6"
server_key_iv: "4ccbe6b8073861c110b2cfe88c5ae158"
server_key_version: "encryptionkey@0"
I20260928 16:43:33.281621 10719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "69ff8cfe1eee4f909cc0157a1eb0b44a"
format_stamp: "Formatted at 2026-09-28 16:43:33 on dist-test-slave-nqgh"
server_key: "73b92ddaa03aacd9e8b1e4d1b94794d6"
server_key_iv: "4ccbe6b8073861c110b2cfe88c5ae158"
server_key_version: "encryptionkey@0"
I20260928 16:43:33.286523 10719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 16:43:33.289922 10734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:33.291904 10719 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260928 16:43:33.292085 10719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "69ff8cfe1eee4f909cc0157a1eb0b44a"
format_stamp: "Formatted at 2026-09-28 16:43:33 on dist-test-slave-nqgh"
server_key: "73b92ddaa03aacd9e8b1e4d1b94794d6"
server_key_iv: "4ccbe6b8073861c110b2cfe88c5ae158"
server_key_version: "encryptionkey@0"
I20260928 16:43:33.292255 10719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:33.310359 10719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:33.314155 10719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:33.314352 10719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:33.315090 10719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:33.316061 10719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:33.316115 10719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:33.316159 10719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:33.316174 10719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:33.327052 10719 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:36217
I20260928 16:43:33.327100 10847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:36217 every 8 connection(s)
I20260928 16:43:33.328177 10719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:43:33.330938 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 10719
I20260928 16:43:33.331100 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:43:33.331454 25885 external_mini_cluster.cc:1428] Setting key 599307f08a1086f3c29bcefb936dbefc
Sep 28 16:43:33 dist-test-slave-nqgh krb5kdc[10478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613813, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.71.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.67@KRBTEST.COM" created.
I20260928 16:43:33.349210 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.331104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:41047 (local address 127.25.71.126:41743)
0928 16:43:33.331304 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:33.331309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:33.333066 (+ 1757us) server_negotiation.cc:407] Connection header received
0928 16:43:33.334527 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:33.334530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:33.334606 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:33.334716 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:33.336316 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.337004 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.339111 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.339484 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.339601 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.343427 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:33.343431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:33.343434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:33.343481 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:33.345468 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:33.346586 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:33.346591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:33.346592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:33.346824 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:33.347422 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:33.347425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:33.347426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:33.347769 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:33.347947 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:33.348761 (+ 814us) server_negotiation.cc:299] Negotiation successful
0928 16:43:33.348987 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 16:43:33.350431 10850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.331642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:41743 (local address 127.25.71.66:41047)
0928 16:43:33.332836 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:33.332924 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:33.334226 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:33.334956 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:33.334964 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:33.335388 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:33.335997 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:33.336030 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.337197 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.337201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:33.338839 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:33.338861 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.339717 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.339723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:33.339995 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.340804 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:33.340831 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:33.343056 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:33.345799 (+ 2743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:33.345807 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:33.345827 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:33.346372 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:33.347072 (+ 700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:33.347075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:33.347077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:33.347284 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:33.347955 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:33.347968 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:33.348422 (+ 454us) client_negotiation.cc:769] Sending connection context
0928 16:43:33.348707 (+ 285us) client_negotiation.cc:240] Negotiation successful
0928 16:43:33.349199 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"thread_start_us":178,"threads_started":1}
I20260928 16:43:33.352020 10848 heartbeater.cc:347] Connected to a master server at 127.25.71.126:41743
I20260928 16:43:33.352425 10848 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:33.353186 10848 heartbeater.cc:511] Master 127.25.71.126:41743 requested a full tablet report, sending...
I20260928 16:43:33.354882 10518 ts_manager.cc:194] Registered new tserver with Master: 69ff8cfe1eee4f909cc0157a1eb0b44a (127.25.71.66:36217)
I20260928 16:43:33.355638 10518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:41047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
I20260928 16:43:33.372056 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41743
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:35295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:33.500705 10853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:33.501040 10853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:33.501088 10853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:33.505774 10853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:33.505873 10853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:33.505937 10853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:43:33.512032 10853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:35295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:41743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:33.513240 10853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:33.514680 10853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:33.526688 10861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:33.526688 10859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:33.526777 10858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:33.527161 10853 server_base.cc:1034] running on GCE node
I20260928 16:43:33.527961 10853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:33.528733 10853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:33.530004 10853 hybrid_clock.cc:648] HybridClock initialized: now 1790613813529975 us; error 55 us; skew 500 ppm
Sep 28 16:43:33 dist-test-slave-nqgh krb5kdc[10478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613813, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:33.533865 10853 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:43:33.535084 10853 webserver.cc:460] Webserver started at http://127.25.71.67:45897/ using document root <none> and password file <none>
I20260928 16:43:33.535699 10853 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:33.535945 10853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:33.536223 10853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:33.538548 10853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "ff8d7206c8174e40b15cc3160f99fe91"
format_stamp: "Formatted at 2026-09-28 16:43:33 on dist-test-slave-nqgh"
server_key: "f41664374adf6aa962eed57c194ea17b"
server_key_iv: "5e3c151923f1d6c537e9333cbe9b6a4d"
server_key_version: "encryptionkey@0"
I20260928 16:43:33.539324 10853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "ff8d7206c8174e40b15cc3160f99fe91"
format_stamp: "Formatted at 2026-09-28 16:43:33 on dist-test-slave-nqgh"
server_key: "f41664374adf6aa962eed57c194ea17b"
server_key_iv: "5e3c151923f1d6c537e9333cbe9b6a4d"
server_key_version: "encryptionkey@0"
I20260928 16:43:33.544116 10853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 16:43:33.547658 10868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:33.549417 10853 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:43:33.549603 10853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "ff8d7206c8174e40b15cc3160f99fe91"
format_stamp: "Formatted at 2026-09-28 16:43:33 on dist-test-slave-nqgh"
server_key: "f41664374adf6aa962eed57c194ea17b"
server_key_iv: "5e3c151923f1d6c537e9333cbe9b6a4d"
server_key_version: "encryptionkey@0"
I20260928 16:43:33.549760 10853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:33.567559 10853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:33.570973 10853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:33.571158 10853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:33.571810 10853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:33.572746 10853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:33.572801 10853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:33.572860 10853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:33.572890 10853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:33.583694 10853 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:35355
I20260928 16:43:33.583809 10982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:35355 every 8 connection(s)
I20260928 16:43:33.584751 10853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:43:33.591482 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 10853
I20260928 16:43:33.591645 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:43:33.591967 25885 external_mini_cluster.cc:1428] Setting key de3c4e1d60f5408348c4ff5633648b51
Sep 28 16:43:33 dist-test-slave-nqgh krb5kdc[10478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613813, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:43:33.602399 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.587658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:52875 (local address 127.25.71.126:41743)
0928 16:43:33.587859 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:33.587864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:33.589250 (+ 1386us) server_negotiation.cc:407] Connection header received
0928 16:43:33.590800 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:33.590803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:33.590869 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:33.590952 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:33.592595 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.593174 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.594500 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.594770 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.594864 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.598091 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:33.598094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:33.598097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:33.598128 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:33.599628 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:33.600453 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:33.600456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:33.600457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:33.600552 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:33.601008 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:33.601010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:33.601011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:33.601218 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:33.601300 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:33.602089 (+ 789us) server_negotiation.cc:299] Negotiation successful
0928 16:43:33.602244 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 16:43:33.603767 10985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.588167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:41743 (local address 127.25.71.67:52875)
0928 16:43:33.589040 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:33.589104 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:33.590524 (+ 1420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:33.591174 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:33.591183 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:33.591716 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:33.592427 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:33.592445 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.593330 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.593333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:33.594340 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:33.594352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.594929 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.594933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:33.595115 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.595700 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:33.595718 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:33.597805 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:33.599815 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:33.599833 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:33.599856 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:33.600319 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:33.600684 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:33.600687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:33.600689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:33.600897 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:33.601345 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:33.601352 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:33.601762 (+ 410us) client_negotiation.cc:769] Sending connection context
0928 16:43:33.601991 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 16:43:33.602411 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":213,"threads_started":1}
I20260928 16:43:33.605649 10983 heartbeater.cc:347] Connected to a master server at 127.25.71.126:41743
I20260928 16:43:33.606024 10983 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:33.606683 10983 heartbeater.cc:511] Master 127.25.71.126:41743 requested a full tablet report, sending...
I20260928 16:43:33.607899 10518 ts_manager.cc:194] Registered new tserver with Master: ff8d7206c8174e40b15cc3160f99fe91 (127.25.71.67:35355)
I20260928 16:43:33.608644 10518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:52875
I20260928 16:43:33.619788 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:43:33.632580 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.622339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55274 (local address 127.25.71.126:41743)
0928 16:43:33.622567 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:33.622581 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:33.622950 (+ 369us) server_negotiation.cc:407] Connection header received
0928 16:43:33.623097 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:33.623099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:33.623156 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:33.623247 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:33.624198 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.624769 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.625893 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.626151 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.626257 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.628201 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:33.628204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:33.628206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:33.628240 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:33.630037 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:33.630883 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:33.630886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:33.630887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:33.631048 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:33.631460 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:33.631463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:33.631464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:33.631755 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:33.631876 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:33.632264 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 16:43:33.632405 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 16:43:33.637120 10518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:43:33.641342 10518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:43:33.660439 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.653231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35220 (local address 127.25.71.65:34379)
0928 16:43:33.653931 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:33.653937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:33.654214 (+ 277us) server_negotiation.cc:407] Connection header received
0928 16:43:33.654327 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:33.654333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:33.654583 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:33.655198 (+ 615us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:33.655365 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.656621 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.658525 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.659501 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.659631 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.659817 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:33.660141 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 16:43:33.660266 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":153,"threads_started":1}
I20260928 16:43:33.660487 10998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.653940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:35355 (local address 127.0.0.1:53792)
0928 16:43:33.654476 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:33.654491 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:33.654576 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:33.655422 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:33.655426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:33.655444 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:33.655581 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:33.655588 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.657016 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.657019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:33.658765 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:33.658781 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.659920 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.659924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:33.660093 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.660163 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 16:43:33.660226 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 16:43:33.660316 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":222,"thread_start_us":116,"threads_started":1}
I20260928 16:43:33.660597 10996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.653231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.25.71.66:36217)
0928 16:43:33.653749 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:33.653761 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:33.654283 (+ 522us) server_negotiation.cc:407] Connection header received
0928 16:43:33.654948 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:33.654956 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:33.655190 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:33.655320 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:33.655784 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.656636 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.658909 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.659723 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.659854 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.660023 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:33.660313 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 16:43:33.660441 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":149,"threads_started":1}
I20260928 16:43:33.660782 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.652906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:34379 (local address 127.0.0.1:35220)
0928 16:43:33.653855 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:33.653928 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:33.654106 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:33.654821 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:33.654828 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:33.654857 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:33.655031 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:33.655039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.656771 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.656775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:33.658364 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:33.658377 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.659683 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.659687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:33.659821 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.659883 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 16:43:33.660017 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 16:43:33.660228 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":76,"thread_start_us":151,"threads_started":1}
I20260928 16:43:33.661113 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.652906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:36217 (local address 127.0.0.1:54388)
0928 16:43:33.653921 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:33.653939 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:33.654736 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:33.655559 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:33.655562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:33.655576 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:33.655667 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:33.655673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.656823 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.656828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:33.658781 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:33.658790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.659956 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.659959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:33.660143 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.660179 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 16:43:33.660869 (+ 690us) client_negotiation.cc:240] Negotiation successful
0928 16:43:33.660970 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"mutex_wait_us":408,"thread_start_us":152,"threads_started":1}
I20260928 16:43:33.661518 10999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.654252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53792 (local address 127.25.71.67:35355)
0928 16:43:33.654955 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:33.654968 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:33.654992 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:43:33.655066 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:33.655071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:33.655257 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:33.655413 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:33.655721 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.656843 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.658941 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.659773 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.660994 (+ 1221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.661186 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:33.661246 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 16:43:33.661355 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":180,"threads_started":1}
I20260928 16:43:33.664619 10648 tablet_service.cc:1467] Processing CreateTablet for tablet 71e0c2163f224af8888c27e5ca2219b8 (DEFAULT_TABLE table=test-table [id=1502f92373d24064a81cf73ae1fd5d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:33.664618 10782 tablet_service.cc:1467] Processing CreateTablet for tablet 71e0c2163f224af8888c27e5ca2219b8 (DEFAULT_TABLE table=test-table [id=1502f92373d24064a81cf73ae1fd5d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:33.664983 10917 tablet_service.cc:1467] Processing CreateTablet for tablet 71e0c2163f224af8888c27e5ca2219b8 (DEFAULT_TABLE table=test-table [id=1502f92373d24064a81cf73ae1fd5d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:33.665910 10782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e0c2163f224af8888c27e5ca2219b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:33.665910 10648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e0c2163f224af8888c27e5ca2219b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:33.666370 10917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e0c2163f224af8888c27e5ca2219b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:33.674399 11000 tablet_bootstrap.cc:492] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace: Bootstrap starting.
I20260928 16:43:33.675203 11001 tablet_bootstrap.cc:492] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91: Bootstrap starting.
I20260928 16:43:33.676133 11002 tablet_bootstrap.cc:492] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a: Bootstrap starting.
I20260928 16:43:33.678265 11000 tablet_bootstrap.cc:654] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:33.678702 11001 tablet_bootstrap.cc:654] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:33.678997 11002 tablet_bootstrap.cc:654] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:33.679495 11000 log.cc:824] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:33.679836 11002 log.cc:824] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:33.679950 11001 log.cc:824] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:33.681988 11000 tablet_bootstrap.cc:492] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace: No bootstrap required, opened a new log
I20260928 16:43:33.682225 11000 ts_tablet_manager.cc:1397] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 16:43:33.682230 11002 tablet_bootstrap.cc:492] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a: No bootstrap required, opened a new log
I20260928 16:43:33.682480 11002 ts_tablet_manager.cc:1397] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:43:33.684139 11001 tablet_bootstrap.cc:492] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91: No bootstrap required, opened a new log
I20260928 16:43:33.684393 11001 ts_tablet_manager.cc:1397] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260928 16:43:33.686858 11002 raft_consensus.cc:348] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ff8cfe1eee4f909cc0157a1eb0b44a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36217 } } peers { permanent_uuid: "475ee1007bfc477284027e09385adace" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34379 } } peers { permanent_uuid: "ff8d7206c8174e40b15cc3160f99fe91" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35355 } }
I20260928 16:43:33.687209 11002 raft_consensus.cc:374] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:33.687275 11002 raft_consensus.cc:729] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ff8cfe1eee4f909cc0157a1eb0b44a, State: Initialized, Role: FOLLOWER
I20260928 16:43:33.687215 11000 raft_consensus.cc:348] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ff8cfe1eee4f909cc0157a1eb0b44a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36217 } } peers { permanent_uuid: "475ee1007bfc477284027e09385adace" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34379 } } peers { permanent_uuid: "ff8d7206c8174e40b15cc3160f99fe91" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35355 } }
I20260928 16:43:33.687547 11000 raft_consensus.cc:374] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:33.687611 11000 raft_consensus.cc:729] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 475ee1007bfc477284027e09385adace, State: Initialized, Role: FOLLOWER
I20260928 16:43:33.687834 11002 consensus_queue.cc:260] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ff8cfe1eee4f909cc0157a1eb0b44a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36217 } } peers { permanent_uuid: "475ee1007bfc477284027e09385adace" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34379 } } peers { permanent_uuid: "ff8d7206c8174e40b15cc3160f99fe91" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35355 } }
I20260928 16:43:33.688103 11000 consensus_queue.cc:260] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ff8cfe1eee4f909cc0157a1eb0b44a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36217 } } peers { permanent_uuid: "475ee1007bfc477284027e09385adace" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34379 } } peers { permanent_uuid: "ff8d7206c8174e40b15cc3160f99fe91" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35355 } }
I20260928 16:43:33.688520 11001 raft_consensus.cc:348] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ff8cfe1eee4f909cc0157a1eb0b44a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36217 } } peers { permanent_uuid: "475ee1007bfc477284027e09385adace" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34379 } } peers { permanent_uuid: "ff8d7206c8174e40b15cc3160f99fe91" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35355 } }
I20260928 16:43:33.688822 11001 raft_consensus.cc:374] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:33.688822 10848 heartbeater.cc:503] Master 127.25.71.126:41743 was elected leader, sending a full tablet report...
I20260928 16:43:33.688875 11001 raft_consensus.cc:729] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8d7206c8174e40b15cc3160f99fe91, State: Initialized, Role: FOLLOWER
I20260928 16:43:33.689235 11002 ts_tablet_manager.cc:1428] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 16:43:33.689284 11000 ts_tablet_manager.cc:1428] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 16:43:33.689343 11001 consensus_queue.cc:260] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ff8cfe1eee4f909cc0157a1eb0b44a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36217 } } peers { permanent_uuid: "475ee1007bfc477284027e09385adace" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34379 } } peers { permanent_uuid: "ff8d7206c8174e40b15cc3160f99fe91" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35355 } }
I20260928 16:43:33.689617 10714 heartbeater.cc:503] Master 127.25.71.126:41743 was elected leader, sending a full tablet report...
I20260928 16:43:33.690398 11001 ts_tablet_manager.cc:1428] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260928 16:43:33.690407 10983 heartbeater.cc:503] Master 127.25.71.126:41743 was elected leader, sending a full tablet report...
W20260928 16:43:33.807240 10715 tablet.cc:2369] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:43:33.829952 10849 tablet.cc:2369] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:43:33.835937 10984 tablet.cc:2369] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:43:33.875371 11007 raft_consensus.cc:482] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:33.875649 11007 raft_consensus.cc:504] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ff8cfe1eee4f909cc0157a1eb0b44a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36217 } } peers { permanent_uuid: "475ee1007bfc477284027e09385adace" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34379 } } peers { permanent_uuid: "ff8d7206c8174e40b15cc3160f99fe91" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35355 } }
I20260928 16:43:33.877070 11007 leader_election.cc:288] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ff8cfe1eee4f909cc0157a1eb0b44a (127.25.71.66:36217), ff8d7206c8174e40b15cc3160f99fe91 (127.25.71.67:35355)
I20260928 16:43:33.883772 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.877601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:36217 (local address 127.25.71.65:42775)
0928 16:43:33.878116 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:33.878135 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:33.878351 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:33.878820 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:33.878823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:33.878847 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:33.879038 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:33.879045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.880226 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.880231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:33.881956 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:33.881972 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.883143 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.883147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:33.883295 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.883340 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 16:43:33.883434 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 16:43:33.883548 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":110,"spinlock_wait_cycles":17408,"thread_start_us":170,"threads_started":1}
I20260928 16:43:33.883842 10999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.877659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:33219 (local address 127.25.71.67:35355)
0928 16:43:33.877912 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:33.877917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:33.878336 (+ 419us) server_negotiation.cc:407] Connection header received
0928 16:43:33.878541 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:33.878546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:33.878620 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:33.878765 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:33.879249 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.880041 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.882206 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.882964 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.883142 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.883211 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:33.883512 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 16:43:33.883654 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 16:43:33.883842 10996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.877746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:42775 (local address 127.25.71.66:36217)
0928 16:43:33.877878 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:33.877883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:33.878301 (+ 418us) server_negotiation.cc:407] Connection header received
0928 16:43:33.878541 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:33.878546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:33.878697 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:33.878782 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:33.879184 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.880064 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.882177 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.882964 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.883142 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.883211 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:33.883512 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 16:43:33.883676 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:43:33.883790 11009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.877601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:35355 (local address 127.25.71.65:33219)
0928 16:43:33.878208 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:33.878221 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:33.878350 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:33.878877 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:33.878880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:33.878894 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:33.879038 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:33.879045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.880226 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.880231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:33.881957 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:33.881972 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.883143 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.883147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:33.883320 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.883362 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 16:43:33.883441 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 16:43:33.883548 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":416,"spinlock_wait_cycles":16000,"thread_start_us":137,"threads_started":1}
I20260928 16:43:33.885046 10937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e0c2163f224af8888c27e5ca2219b8" candidate_uuid: "475ee1007bfc477284027e09385adace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8d7206c8174e40b15cc3160f99fe91" is_pre_election: true
I20260928 16:43:33.885175 11006 raft_consensus.cc:482] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:33.885108 10802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e0c2163f224af8888c27e5ca2219b8" candidate_uuid: "475ee1007bfc477284027e09385adace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ff8cfe1eee4f909cc0157a1eb0b44a" is_pre_election: true
I20260928 16:43:33.885425 10937 raft_consensus.cc:2454] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 475ee1007bfc477284027e09385adace in term 0.
I20260928 16:43:33.885389 11006 raft_consensus.cc:504] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ff8cfe1eee4f909cc0157a1eb0b44a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36217 } } peers { permanent_uuid: "475ee1007bfc477284027e09385adace" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34379 } } peers { permanent_uuid: "ff8d7206c8174e40b15cc3160f99fe91" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35355 } }
I20260928 16:43:33.886013 10802 raft_consensus.cc:2454] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 475ee1007bfc477284027e09385adace in term 0.
I20260928 16:43:33.886102 10601 leader_election.cc:302] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 475ee1007bfc477284027e09385adace, ff8d7206c8174e40b15cc3160f99fe91; no voters:
I20260928 16:43:33.886404 11007 raft_consensus.cc:2781] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:43:33.886498 11007 raft_consensus.cc:482] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:43:33.886543 11007 raft_consensus.cc:3037] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:33.886916 11006 leader_election.cc:288] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 475ee1007bfc477284027e09385adace (127.25.71.65:34379), ff8d7206c8174e40b15cc3160f99fe91 (127.25.71.67:35355)
I20260928 16:43:33.888105 11007 raft_consensus.cc:504] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ff8cfe1eee4f909cc0157a1eb0b44a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36217 } } peers { permanent_uuid: "475ee1007bfc477284027e09385adace" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34379 } } peers { permanent_uuid: "ff8d7206c8174e40b15cc3160f99fe91" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35355 } }
I20260928 16:43:33.888592 11007 leader_election.cc:288] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace [CANDIDATE]: Term 1 election: Requested vote from peers 69ff8cfe1eee4f909cc0157a1eb0b44a (127.25.71.66:36217), ff8d7206c8174e40b15cc3160f99fe91 (127.25.71.67:35355)
I20260928 16:43:33.889339 10937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e0c2163f224af8888c27e5ca2219b8" candidate_uuid: "475ee1007bfc477284027e09385adace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8d7206c8174e40b15cc3160f99fe91"
I20260928 16:43:33.889605 10937 raft_consensus.cc:3037] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:33.889894 10802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e0c2163f224af8888c27e5ca2219b8" candidate_uuid: "475ee1007bfc477284027e09385adace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ff8cfe1eee4f909cc0157a1eb0b44a"
I20260928 16:43:33.890074 10802 raft_consensus.cc:3037] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:33.891746 10802 raft_consensus.cc:2454] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 475ee1007bfc477284027e09385adace in term 1.
I20260928 16:43:33.891891 10937 raft_consensus.cc:2454] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 475ee1007bfc477284027e09385adace in term 1.
I20260928 16:43:33.892323 10601 leader_election.cc:302] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 475ee1007bfc477284027e09385adace, ff8d7206c8174e40b15cc3160f99fe91; no voters:
I20260928 16:43:33.892717 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.887242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:45555 (local address 127.25.71.65:34379)
0928 16:43:33.887399 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:33.887404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:33.887559 (+ 155us) server_negotiation.cc:407] Connection header received
0928 16:43:33.887809 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:33.887813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:33.887876 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:33.887944 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:33.888563 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.889235 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.891086 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.891819 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.891935 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.891996 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:33.892463 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 16:43:33.892564 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:43:33.892917 11007 raft_consensus.cc:2781] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:33.893138 10999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.887685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:46223 (local address 127.25.71.67:35355)
0928 16:43:33.887864 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:33.887869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:33.888221 (+ 352us) server_negotiation.cc:407] Connection header received
0928 16:43:33.888414 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:33.888417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:33.888484 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:33.888578 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:33.889264 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.890076 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.891869 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.892467 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.892554 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.892610 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:33.892925 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 16:43:33.893007 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:43:33.893177 11011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.887149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:34379 (local address 127.25.71.66:45555)
0928 16:43:33.887451 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:33.887467 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:33.887659 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:33.888153 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:33.888156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:33.888179 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:33.888424 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:33.888432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.889374 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.889378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:33.890914 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:33.890928 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.892192 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.892195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:33.892307 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.892333 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:43:33.892926 (+ 593us) client_negotiation.cc:240] Negotiation successful
0928 16:43:33.893013 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":28,"thread_start_us":103,"threads_started":1}
I20260928 16:43:33.893528 11012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.887549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:35355 (local address 127.25.71.66:46223)
0928 16:43:33.888069 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:33.888086 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:33.888231 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:33.888935 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:33.888939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:33.888969 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:33.889140 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:33.889148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.890366 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.890370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:33.891716 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:33.891729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.892703 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.892706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:33.892791 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.892812 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 16:43:33.893298 (+ 486us) client_negotiation.cc:240] Negotiation successful
0928 16:43:33.893381 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":1,"spinlock_wait_cycles":40192,"thread_start_us":78,"threads_started":1}
I20260928 16:43:33.893633 11007 raft_consensus.cc:686] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace [term 1 LEADER]: Becoming Leader. State: Replica: 475ee1007bfc477284027e09385adace, State: Running, Role: LEADER
I20260928 16:43:33.893963 10668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e0c2163f224af8888c27e5ca2219b8" candidate_uuid: "69ff8cfe1eee4f909cc0157a1eb0b44a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "475ee1007bfc477284027e09385adace" is_pre_election: true
I20260928 16:43:33.894496 11007 consensus_queue.cc:237] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ff8cfe1eee4f909cc0157a1eb0b44a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36217 } } peers { permanent_uuid: "475ee1007bfc477284027e09385adace" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34379 } } peers { permanent_uuid: "ff8d7206c8174e40b15cc3160f99fe91" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35355 } }
I20260928 16:43:33.895246 10937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e0c2163f224af8888c27e5ca2219b8" candidate_uuid: "69ff8cfe1eee4f909cc0157a1eb0b44a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8d7206c8174e40b15cc3160f99fe91" is_pre_election: true
I20260928 16:43:33.895503 10937 raft_consensus.cc:2379] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69ff8cfe1eee4f909cc0157a1eb0b44a in current term 1: Already voted for candidate 475ee1007bfc477284027e09385adace in this term.
I20260928 16:43:33.896189 10735 leader_election.cc:302] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ff8cfe1eee4f909cc0157a1eb0b44a; no voters: 475ee1007bfc477284027e09385adace, ff8d7206c8174e40b15cc3160f99fe91
I20260928 16:43:33.896632 11006 raft_consensus.cc:2726] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 16:43:33.899998 10519 catalog_manager.cc:5495] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace reported cstate change: term changed from 0 to 1, leader changed from <none> to 475ee1007bfc477284027e09385adace (127.25.71.65). New cstate: current_term: 1 leader_uuid: "475ee1007bfc477284027e09385adace" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ff8cfe1eee4f909cc0157a1eb0b44a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "475ee1007bfc477284027e09385adace" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff8d7206c8174e40b15cc3160f99fe91" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35355 } health_report { overall_health: UNKNOWN } } }
I20260928 16:43:33.964468 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.960302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35232 (local address 127.25.71.65:34379)
0928 16:43:33.960454 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:33.960458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:33.960473 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:43:33.960571 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:33.960573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:33.960644 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:33.960746 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:33.961197 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.961731 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.962948 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.963192 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.963285 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.963591 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:33.964082 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:33.964179 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:33.964283 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 16:43:33.964345 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:43:33.972168 10937 raft_consensus.cc:1261] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91 [term 1 FOLLOWER]: Refusing update from remote peer 475ee1007bfc477284027e09385adace: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:33.972241 10802 raft_consensus.cc:1261] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a [term 1 FOLLOWER]: Refusing update from remote peer 475ee1007bfc477284027e09385adace: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:33.973084 11013 consensus_queue.cc:1035] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8d7206c8174e40b15cc3160f99fe91" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:33.973384 11007 consensus_queue.cc:1035] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ff8cfe1eee4f909cc0157a1eb0b44a" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:33.983093 11017 mvcc.cc:205] Tried to move back new op lower bound from 7334354182019862528 to 7334354181719949312. Current Snapshot: MvccSnapshot[applied={T|T < 7334354182019862528}]
I20260928 16:43:33.984211 11016 mvcc.cc:205] Tried to move back new op lower bound from 7334354182019862528 to 7334354181719949312. Current Snapshot: MvccSnapshot[applied={T|T < 7334354182019862528}]
I20260928 16:43:33.995234 10520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:33.995489 10520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:33.998991 10520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7f62471597e46f689d17928a7f5eda1: Sending DeleteTablet for 3 replicas of tablet 71e0c2163f224af8888c27e5ca2219b8
I20260928 16:43:34.000195 10917 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e0c2163f224af8888c27e5ca2219b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:33 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:53792
I20260928 16:43:34.000203 10782 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e0c2163f224af8888c27e5ca2219b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:33 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:54388
I20260928 16:43:34.000211 10648 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e0c2163f224af8888c27e5ca2219b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:33 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:35220
I20260928 16:43:34.000916 11027 tablet_replica.cc:330] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace: stopping tablet replica
I20260928 16:43:34.001174 11026 tablet_replica.cc:330] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91: stopping tablet replica
I20260928 16:43:34.001327 11027 raft_consensus.cc:2229] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:43:34.001608 11026 raft_consensus.cc:2229] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:34.001825 11027 raft_consensus.cc:2258] T 71e0c2163f224af8888c27e5ca2219b8 P 475ee1007bfc477284027e09385adace [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:34.002041 11026 raft_consensus.cc:2258] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:34.002120 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 10585
I20260928 16:43:34.002276 11028 tablet_replica.cc:330] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a: stopping tablet replica
I20260928 16:43:34.004057 11028 raft_consensus.cc:2229] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:34.004428 11028 raft_consensus.cc:2258] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:34.005363 11026 ts_tablet_manager.cc:1905] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:34.006203 11028 ts_tablet_manager.cc:1905] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:34.008705 11026 ts_tablet_manager.cc:1918] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:43:34.008836 11026 log.cc:1196] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/71e0c2163f224af8888c27e5ca2219b8
I20260928 16:43:34.009171 11026 ts_tablet_manager.cc:1939] T 71e0c2163f224af8888c27e5ca2219b8 P ff8d7206c8174e40b15cc3160f99fe91: Deleting consensus metadata
I20260928 16:43:34.010088 11028 ts_tablet_manager.cc:1918] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:43:34.010227 11028 log.cc:1196] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/71e0c2163f224af8888c27e5ca2219b8
I20260928 16:43:34.010535 11028 ts_tablet_manager.cc:1939] T 71e0c2163f224af8888c27e5ca2219b8 P 69ff8cfe1eee4f909cc0157a1eb0b44a: Deleting consensus metadata
I20260928 16:43:34.010847 10507 catalog_manager.cc:4841] TS ff8d7206c8174e40b15cc3160f99fe91 (127.25.71.67:35355): tablet 71e0c2163f224af8888c27e5ca2219b8 (table test-table [id=1502f92373d24064a81cf73ae1fd5d4d]) successfully deleted
W20260928 16:43:34.011268 10507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.71.65:34379 (error 108)
I20260928 16:43:34.011571 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 10719
I20260928 16:43:34.012712 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:34.012387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:34379 (local address 127.0.0.1:35248)
0928 16:43:34.012534 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:34.012615 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.65:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 16:43:34.012919 10507 catalog_manager.cc:4584] TS 475ee1007bfc477284027e09385adace (127.25.71.65:34379): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71e0c2163f224af8888c27e5ca2219b8: Network error: Client connection negotiation failed: client connection to 127.25.71.65:34379: connect: Connection refused (error 111)
I20260928 16:43:34.017686 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 10853
I20260928 16:43:34.019682 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:34.017987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:36217 (local address 127.0.0.1:54398)
0928 16:43:34.019488 (+ 1501us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:34.019563 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.66:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1401}
I20260928 16:43:34.024718 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 10491
2026-09-28T16:43:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2626 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T16:43:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:34Z Disabled control of system clock
I20260928 16:43:34.059228 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:35369
--webserver_interface=127.25.71.126
--webserver_port=0
--builtin_ntp_servers=127.25.71.84:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:35369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:43:34.197970 11033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:34.198375 11033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:34.198417 11033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:34.203056 11033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:34.203166 11033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:34.203179 11033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:34.203197 11033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:34.203218 11033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:34.208920 11033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:35369
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:35369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:34.210548 11033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:34.212399 11033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:34.223259 11041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:34.223429 11033 server_base.cc:1034] running on GCE node
W20260928 16:43:34.223265 11038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:34.223259 11039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:34.224572 11033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:34.225941 11033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:34.227335 11033 hybrid_clock.cc:648] HybridClock initialized: now 1790613814227303 us; error 65 us; skew 500 ppm
I20260928 16:43:34.229781 11033 webserver.cc:460] Webserver started at http://127.25.71.126:42701/ using document root <none> and password file <none>
I20260928 16:43:34.230490 11033 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:34.230569 11033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:34.230837 11033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:34.233186 11033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "241c3e1cff20455790169b17e2c0ff33"
format_stamp: "Formatted at 2026-09-28 16:43:34 on dist-test-slave-nqgh"
server_key: "4ee62564811efaf4fcc2eee49194f2c4"
server_key_iv: "d1069c716d0d648fa32e4f7015ee27f0"
server_key_version: "encryptionkey@0"
I20260928 16:43:34.233840 11033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "241c3e1cff20455790169b17e2c0ff33"
format_stamp: "Formatted at 2026-09-28 16:43:34 on dist-test-slave-nqgh"
server_key: "4ee62564811efaf4fcc2eee49194f2c4"
server_key_iv: "d1069c716d0d648fa32e4f7015ee27f0"
server_key_version: "encryptionkey@0"
I20260928 16:43:34.238534 11033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:43:34.241914 11047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:34.243815 11033 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260928 16:43:34.244014 11033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "241c3e1cff20455790169b17e2c0ff33"
format_stamp: "Formatted at 2026-09-28 16:43:34 on dist-test-slave-nqgh"
server_key: "4ee62564811efaf4fcc2eee49194f2c4"
server_key_iv: "d1069c716d0d648fa32e4f7015ee27f0"
server_key_version: "encryptionkey@0"
I20260928 16:43:34.244158 11033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:34.259171 11033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:34.259979 11033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:34.260185 11033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:34.268155 11033 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:35369
I20260928 16:43:34.268189 11099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:35369 every 8 connection(s)
I20260928 16:43:34.269243 11033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:34.273330 11100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:34.278366 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 11033
I20260928 16:43:34.278522 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:43:34.278906 25885 external_mini_cluster.cc:1428] Setting key 64cc0f4eab34d0ded6e8c4cebbbed8ee
I20260928 16:43:34.281126 11100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33: Bootstrap starting.
I20260928 16:43:34.284554 11100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:34.285607 11100 log.cc:824] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:34.288759 11100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33: No bootstrap required, opened a new log
I20260928 16:43:34.290787 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:34.280759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41046 (local address 127.25.71.126:35369)
0928 16:43:34.281654 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:34.281678 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:34.281722 (+ 44us) server_negotiation.cc:407] Connection header received
0928 16:43:34.282608 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:34.282692 (+ 84us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:34.283212 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:34.283556 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:34.284221 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:34.285345 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:34.286676 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:34.287310 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:34.287484 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:34.288535 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:34.288541 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:34.288557 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:34.289044 (+ 487us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:34.289120 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:34.289144 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:34.289236 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:34.289498 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 16:43:34.289853 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":150,"threads_started":1}
I20260928 16:43:34.293680 11100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241c3e1cff20455790169b17e2c0ff33" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35369 } }
I20260928 16:43:34.294070 11100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:34.294138 11100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 241c3e1cff20455790169b17e2c0ff33, State: Initialized, Role: FOLLOWER
I20260928 16:43:34.294652 11100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241c3e1cff20455790169b17e2c0ff33" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35369 } }
I20260928 16:43:34.294806 11100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:34.294852 11100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:34.294930 11100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:34.296142 11100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241c3e1cff20455790169b17e2c0ff33" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35369 } }
I20260928 16:43:34.296550 11100 leader_election.cc:302] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 241c3e1cff20455790169b17e2c0ff33; no voters:
I20260928 16:43:34.296917 11100 leader_election.cc:288] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:34.297173 11105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:34.297567 11105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33 [term 1 LEADER]: Becoming Leader. State: Replica: 241c3e1cff20455790169b17e2c0ff33, State: Running, Role: LEADER
I20260928 16:43:34.298038 11105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241c3e1cff20455790169b17e2c0ff33" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35369 } }
I20260928 16:43:34.298576 11100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:34.300096 11107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 241c3e1cff20455790169b17e2c0ff33. Latest consensus state: current_term: 1 leader_uuid: "241c3e1cff20455790169b17e2c0ff33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241c3e1cff20455790169b17e2c0ff33" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35369 } } }
I20260928 16:43:34.300079 11106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "241c3e1cff20455790169b17e2c0ff33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241c3e1cff20455790169b17e2c0ff33" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35369 } } }
I20260928 16:43:34.300287 11106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:34.300601 11107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33 [sys.catalog]: This master's current role is: LEADER
W20260928 16:43:34.304781 11120 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:43:34.304919 11120 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:43:34.304989 11114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:34.305897 11114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:34.314100 11114 catalog_manager.cc:1347] Generated new cluster ID: 43ca4f2cf23242e4911b9804e129134e
I20260928 16:43:34.314239 11114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:34.327201 11114 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:34.328855 11114 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:34.336143 11114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33: Generated new TSK 0
I20260928 16:43:34.337060 11114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:43:34.341146 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:35369
--builtin_ntp_servers=127.25.71.84:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:43:34.467561 11124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:34.467875 11124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:34.467916 11124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:34.472196 11124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:34.472277 11124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:34.472337 11124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:43:34.477499 11124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:35369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:34.478726 11124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:34.480227 11124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:34.491091 11129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:34.491322 11130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:34.491403 11124 server_base.cc:1034] running on GCE node
W20260928 16:43:34.491460 11132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:34.492093 11124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:34.493170 11124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:34.494428 11124 hybrid_clock.cc:648] HybridClock initialized: now 1790613814494364 us; error 83 us; skew 500 ppm
I20260928 16:43:34.497452 11124 webserver.cc:460] Webserver started at http://127.25.71.65:46503/ using document root <none> and password file <none>
I20260928 16:43:34.498346 11124 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:34.498454 11124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:34.498908 11124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:34.502115 11124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "595abd90c2fd4d9c9575eaaf70cd3c19"
format_stamp: "Formatted at 2026-09-28 16:43:34 on dist-test-slave-nqgh"
server_key: "e8495da602597b271e8c22ca53beac0f"
server_key_iv: "b20c7a24a2fffcc810864a55d9e5ec03"
server_key_version: "encryptionkey@0"
I20260928 16:43:34.502985 11124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "595abd90c2fd4d9c9575eaaf70cd3c19"
format_stamp: "Formatted at 2026-09-28 16:43:34 on dist-test-slave-nqgh"
server_key: "e8495da602597b271e8c22ca53beac0f"
server_key_iv: "b20c7a24a2fffcc810864a55d9e5ec03"
server_key_version: "encryptionkey@0"
I20260928 16:43:34.508240 11124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:43:34.512250 11138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:34.514088 11124 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:43:34.514261 11124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "595abd90c2fd4d9c9575eaaf70cd3c19"
format_stamp: "Formatted at 2026-09-28 16:43:34 on dist-test-slave-nqgh"
server_key: "e8495da602597b271e8c22ca53beac0f"
server_key_iv: "b20c7a24a2fffcc810864a55d9e5ec03"
server_key_version: "encryptionkey@0"
I20260928 16:43:34.514438 11124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:34.532649 11124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:34.533592 11124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:34.533825 11124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:34.534615 11124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:34.535914 11124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:34.535971 11124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:34.536046 11124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:34.536070 11124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:34.547483 11124 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:41045
I20260928 16:43:34.547595 11251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:41045 every 8 connection(s)
I20260928 16:43:34.548894 11124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:43:34.550267 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 11124
I20260928 16:43:34.550426 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:43:34.550870 25885 external_mini_cluster.cc:1428] Setting key c263778c2873510d34a608e079948625
I20260928 16:43:34.553764 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:35369
--builtin_ntp_servers=127.25.71.84:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:43:34.560304 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:34.551731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:55523 (local address 127.25.71.126:35369)
0928 16:43:34.551918 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:34.551923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:34.553501 (+ 1578us) server_negotiation.cc:407] Connection header received
0928 16:43:34.555122 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:34.555125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:34.555184 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:34.555258 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:34.556274 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:34.556834 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:34.558148 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:34.558419 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:34.558517 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:34.559811 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:34.559815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:34.559818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:34.559832 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:34.559852 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:34.559860 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:34.559921 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:34.560070 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 16:43:34.560141 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:43:34.561254 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:34.552328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35369 (local address 127.25.71.65:55523)
0928 16:43:34.553295 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:34.553352 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:34.554866 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:34.555457 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:34.555465 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:34.555560 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:34.556118 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:34.556131 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:34.556973 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:34.556976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:34.557918 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:34.557991 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:34.558594 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:34.558599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:34.558816 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:34.559519 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:43:34.559521 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:34.559565 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:43:34.559639 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:34.559932 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:34.559941 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:34.559947 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:43:34.560061 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 16:43:34.560368 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":173,"threads_started":1}
I20260928 16:43:34.562887 11252 heartbeater.cc:347] Connected to a master server at 127.25.71.126:35369
I20260928 16:43:34.563162 11252 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:34.563769 11252 heartbeater.cc:511] Master 127.25.71.126:35369 requested a full tablet report, sending...
I20260928 16:43:34.565593 11064 ts_manager.cc:194] Registered new tserver with Master: 595abd90c2fd4d9c9575eaaf70cd3c19 (127.25.71.65:41045)
I20260928 16:43:34.566937 11064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.71.65:55523
W20260928 16:43:34.693051 11255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:34.693499 11255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:34.693553 11255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:34.698237 11255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:34.698371 11255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:34.698443 11255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:43:34.704389 11255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:35369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:34.705917 11255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:34.707801 11255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:34.720438 11263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:34.720572 11255 server_base.cc:1034] running on GCE node
W20260928 16:43:34.720455 11260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:34.720438 11261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:34.721433 11255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:34.722332 11255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:34.723520 11255 hybrid_clock.cc:648] HybridClock initialized: now 1790613814723447 us; error 122 us; skew 500 ppm
I20260928 16:43:34.726022 11255 webserver.cc:460] Webserver started at http://127.25.71.66:46521/ using document root <none> and password file <none>
I20260928 16:43:34.726799 11255 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:34.726877 11255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:34.727116 11255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:34.729729 11255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "ba146c9f56cd4a8aa16f6af2125eb2dd"
format_stamp: "Formatted at 2026-09-28 16:43:34 on dist-test-slave-nqgh"
server_key: "1c65cef9129815ac673eb5cd3c8da217"
server_key_iv: "62af403b125896c026d98958a21508d0"
server_key_version: "encryptionkey@0"
I20260928 16:43:34.730400 11255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "ba146c9f56cd4a8aa16f6af2125eb2dd"
format_stamp: "Formatted at 2026-09-28 16:43:34 on dist-test-slave-nqgh"
server_key: "1c65cef9129815ac673eb5cd3c8da217"
server_key_iv: "62af403b125896c026d98958a21508d0"
server_key_version: "encryptionkey@0"
I20260928 16:43:34.735544 11255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 16:43:34.739006 11269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:34.740728 11255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:43:34.740896 11255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "ba146c9f56cd4a8aa16f6af2125eb2dd"
format_stamp: "Formatted at 2026-09-28 16:43:34 on dist-test-slave-nqgh"
server_key: "1c65cef9129815ac673eb5cd3c8da217"
server_key_iv: "62af403b125896c026d98958a21508d0"
server_key_version: "encryptionkey@0"
I20260928 16:43:34.741067 11255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:34.753723 11255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:34.754580 11255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:34.754805 11255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:34.755496 11255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:34.756671 11255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:34.756721 11255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:34.756767 11255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:34.756788 11255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:34.768488 11255 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:37097
I20260928 16:43:34.768532 11382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:37097 every 8 connection(s)
I20260928 16:43:34.769641 11255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:43:34.772794 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 11255
I20260928 16:43:34.772938 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:43:34.773312 25885 external_mini_cluster.cc:1428] Setting key 364fe4d338b23f864d149fe716a7883d
I20260928 16:43:34.776223 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:35369
--builtin_ntp_servers=127.25.71.84:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:43:34.781407 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:34.771872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:48869 (local address 127.25.71.126:35369)
0928 16:43:34.772022 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:34.772027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:34.773560 (+ 1533us) server_negotiation.cc:407] Connection header received
0928 16:43:34.775519 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:34.775523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:34.775574 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:34.775658 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:34.777238 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:34.777828 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:34.779309 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:34.779543 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:34.779636 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:34.780895 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:34.780898 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:34.780900 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:34.780915 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:34.780936 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:34.780943 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:34.781003 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:34.781160 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 16:43:34.781233 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:43:34.782361 11385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:34.772223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35369 (local address 127.25.71.66:48869)
0928 16:43:34.773318 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:34.773394 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:34.775232 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:34.775918 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:34.775929 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:34.776087 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:34.777010 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:34.777032 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:34.778052 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:34.778056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:34.779159 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:34.779175 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:34.779715 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:34.779719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:34.779915 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:34.780583 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:43:34.780585 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:34.780634 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:43:34.780716 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:34.781016 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:34.781026 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:34.781034 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:43:34.781149 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 16:43:34.781475 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":149,"threads_started":1}
I20260928 16:43:34.784099 11383 heartbeater.cc:347] Connected to a master server at 127.25.71.126:35369
I20260928 16:43:34.784400 11383 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:34.785073 11383 heartbeater.cc:511] Master 127.25.71.126:35369 requested a full tablet report, sending...
I20260928 16:43:34.786463 11064 ts_manager.cc:194] Registered new tserver with Master: ba146c9f56cd4a8aa16f6af2125eb2dd (127.25.71.66:37097)
I20260928 16:43:34.787148 11064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.71.66:48869
W20260928 16:43:34.914598 11386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:34.914994 11386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:34.915059 11386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:34.919682 11386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:34.919783 11386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:34.919873 11386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:43:34.925649 11386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:35369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:34.927292 11386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:34.929229 11386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:34.942055 11392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:34.942055 11394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:34.942080 11391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:34.942559 11386 server_base.cc:1034] running on GCE node
I20260928 16:43:34.943138 11386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:34.943933 11386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:34.945164 11386 hybrid_clock.cc:648] HybridClock initialized: now 1790613814945136 us; error 65 us; skew 500 ppm
I20260928 16:43:34.947700 11386 webserver.cc:460] Webserver started at http://127.25.71.67:34177/ using document root <none> and password file <none>
I20260928 16:43:34.948480 11386 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:34.948563 11386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:34.948789 11386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:34.951256 11386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "1944b8253eab45638b870e7314af5085"
format_stamp: "Formatted at 2026-09-28 16:43:34 on dist-test-slave-nqgh"
server_key: "e13d0c0ee0c18e0e0a8089a07c2ec751"
server_key_iv: "9c25e7ed2f6aa20e712bdd71bd768c15"
server_key_version: "encryptionkey@0"
I20260928 16:43:34.951889 11386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "1944b8253eab45638b870e7314af5085"
format_stamp: "Formatted at 2026-09-28 16:43:34 on dist-test-slave-nqgh"
server_key: "e13d0c0ee0c18e0e0a8089a07c2ec751"
server_key_iv: "9c25e7ed2f6aa20e712bdd71bd768c15"
server_key_version: "encryptionkey@0"
I20260928 16:43:34.956681 11386 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:43:34.959897 11400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:34.961674 11386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:43:34.961845 11386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "1944b8253eab45638b870e7314af5085"
format_stamp: "Formatted at 2026-09-28 16:43:34 on dist-test-slave-nqgh"
server_key: "e13d0c0ee0c18e0e0a8089a07c2ec751"
server_key_iv: "9c25e7ed2f6aa20e712bdd71bd768c15"
server_key_version: "encryptionkey@0"
I20260928 16:43:34.961973 11386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:34.974858 11386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:34.975646 11386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:34.975844 11386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:34.976500 11386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:34.977576 11386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:34.977628 11386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:34.977686 11386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:34.977715 11386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:34.988970 11386 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:43455
I20260928 16:43:34.989035 11513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:43455 every 8 connection(s)
I20260928 16:43:34.990115 11386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:43:34.995460 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 11386
I20260928 16:43:34.995610 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:43:34.995958 25885 external_mini_cluster.cc:1428] Setting key cb172624caeba42420aaa38a5604ed7b
I20260928 16:43:35.003983 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:34.992814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:49999 (local address 127.25.71.126:35369)
0928 16:43:34.993064 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:34.993070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:34.994900 (+ 1830us) server_negotiation.cc:407] Connection header received
0928 16:43:34.997226 (+ 2326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:34.997229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:34.997304 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:34.997416 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:34.998995 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:34.999707 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.001347 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.001616 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.001713 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:35.003335 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:35.003339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:35.003341 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:35.003358 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:35.003385 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:35.003394 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:35.003510 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:35.003751 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 16:43:35.003826 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260928 16:43:35.005198 11516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:34.993473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35369 (local address 127.25.71.67:49999)
0928 16:43:34.994532 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:34.994597 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:34.996802 (+ 2205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:34.997733 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:34.997746 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:34.997886 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:34.998786 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:34.998807 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:34.999888 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:34.999892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:35.001168 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:35.001183 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.001807 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.001811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:35.002018 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:35.002939 (+ 921us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:43:35.002941 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:35.003001 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:43:35.003107 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:35.003527 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:35.003536 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:35.003541 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:43:35.003676 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 16:43:35.004011 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":159,"threads_started":1}
I20260928 16:43:35.006840 11514 heartbeater.cc:347] Connected to a master server at 127.25.71.126:35369
I20260928 16:43:35.007211 11514 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:35.007902 11514 heartbeater.cc:511] Master 127.25.71.126:35369 requested a full tablet report, sending...
I20260928 16:43:35.009298 11064 ts_manager.cc:194] Registered new tserver with Master: 1944b8253eab45638b870e7314af5085 (127.25.71.67:43455)
I20260928 16:43:35.009967 11064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.71.67:49999
I20260928 16:43:35.011572 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:43:35.018936 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:35.014368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54714 (local address 127.25.71.126:35369)
0928 16:43:35.014561 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:35.014565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:35.014717 (+ 152us) server_negotiation.cc:407] Connection header received
0928 16:43:35.014890 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:35.014892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:35.014942 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:35.015022 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:35.015356 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.015937 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.016979 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.017216 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.017310 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:35.018288 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:35.018292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:35.018294 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:35.018309 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:35.018369 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:35.018381 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:35.018461 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:35.018698 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 16:43:35.018769 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 16:43:35.023221 11064 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:43:35.026593 11064 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:43:35.047077 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:35.039101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:41045 (local address 127.0.0.1:47550)
0928 16:43:35.040405 (+ 1304us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:35.040480 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:35.040617 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:35.041407 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:35.041414 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:35.041436 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:35.041600 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:35.041607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.042936 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.042940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:35.045067 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:35.045080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.046602 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.046606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:35.046741 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:35.046764 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:43:35.046811 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 16:43:35.046895 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"mutex_wait_us":125,"thread_start_us":99,"threads_started":1}
I20260928 16:43:35.047081 11529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:35.040145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33466 (local address 127.25.71.66:37097)
0928 16:43:35.041070 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:35.041081 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:35.041106 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:43:35.041182 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:35.041186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:35.041344 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:35.041484 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:35.041933 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.043271 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.044769 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.045569 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.045759 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:35.046049 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:35.046698 (+ 649us) server_negotiation.cc:299] Negotiation successful
0928 16:43:35.046858 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":812,"thread_start_us":130,"threads_started":1}
I20260928 16:43:35.047115 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:35.039475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52212 (local address 127.25.71.67:43455)
0928 16:43:35.040123 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:35.040139 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:35.040619 (+ 480us) server_negotiation.cc:407] Connection header received
0928 16:43:35.040900 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:35.040908 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:35.041204 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:35.041365 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:35.041911 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.043271 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.045275 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.045991 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.046109 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:35.046358 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:35.046785 (+ 427us) server_negotiation.cc:299] Negotiation successful
0928 16:43:35.046915 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":195,"threads_started":1}
I20260928 16:43:35.047712 11525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:35.039347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:37097 (local address 127.0.0.1:33466)
0928 16:43:35.040792 (+ 1445us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:35.040807 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:35.040893 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:35.041595 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:35.041598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:35.041638 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:35.041804 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:35.041811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.043457 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.043461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:35.044638 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:35.044651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.046141 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.046145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:35.046348 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:35.046422 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 16:43:35.046517 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 16:43:35.047561 (+ 1044us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1361,"mutex_wait_us":1097,"spinlock_wait_cycles":8192,"thread_start_us":158,"threads_started":1}
I20260928 16:43:35.048018 11527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:35.039119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:43455 (local address 127.0.0.1:52212)
0928 16:43:35.040405 (+ 1286us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:35.040507 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:35.040725 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:35.041594 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:35.041597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:35.041633 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:35.041777 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:35.041783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.043409 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.043412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:35.045067 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:35.045080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.046135 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.046139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:35.046312 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:35.046396 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 16:43:35.046510 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 16:43:35.047889 (+ 1379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"mutex_wait_us":121,"spinlock_wait_cycles":3968,"thread_start_us":109,"threads_started":1}
I20260928 16:43:35.049814 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:35.040575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47550 (local address 127.25.71.65:41045)
0928 16:43:35.040952 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:35.040959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:35.040988 (+ 29us) server_negotiation.cc:407] Connection header received
0928 16:43:35.041059 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:35.041064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:35.041240 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:35.041394 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:35.041735 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.042567 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.045239 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.046081 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.048979 (+ 2898us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:35.049381 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:35.049453 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 16:43:35.049602 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":105,"threads_started":1}
I20260928 16:43:35.051432 11317 tablet_service.cc:1467] Processing CreateTablet for tablet 65abd25638c44e2ba28089d387a6f88d (DEFAULT_TABLE table=test-table [id=310ee05de97a4674880cc043005ef86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:35.051781 11448 tablet_service.cc:1467] Processing CreateTablet for tablet 65abd25638c44e2ba28089d387a6f88d (DEFAULT_TABLE table=test-table [id=310ee05de97a4674880cc043005ef86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:35.052717 11317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65abd25638c44e2ba28089d387a6f88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:35.053126 11448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65abd25638c44e2ba28089d387a6f88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:35.053359 11186 tablet_service.cc:1467] Processing CreateTablet for tablet 65abd25638c44e2ba28089d387a6f88d (DEFAULT_TABLE table=test-table [id=310ee05de97a4674880cc043005ef86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:35.054553 11186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65abd25638c44e2ba28089d387a6f88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:35.059937 11531 tablet_bootstrap.cc:492] T 65abd25638c44e2ba28089d387a6f88d P ba146c9f56cd4a8aa16f6af2125eb2dd: Bootstrap starting.
I20260928 16:43:35.062319 11532 tablet_bootstrap.cc:492] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085: Bootstrap starting.
I20260928 16:43:35.063441 11533 tablet_bootstrap.cc:492] T 65abd25638c44e2ba28089d387a6f88d P 595abd90c2fd4d9c9575eaaf70cd3c19: Bootstrap starting.
I20260928 16:43:35.063616 11531 tablet_bootstrap.cc:654] T 65abd25638c44e2ba28089d387a6f88d P ba146c9f56cd4a8aa16f6af2125eb2dd: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:35.064867 11531 log.cc:824] T 65abd25638c44e2ba28089d387a6f88d P ba146c9f56cd4a8aa16f6af2125eb2dd: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:35.065182 11532 tablet_bootstrap.cc:654] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:35.065950 11533 tablet_bootstrap.cc:654] T 65abd25638c44e2ba28089d387a6f88d P 595abd90c2fd4d9c9575eaaf70cd3c19: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:35.066005 11532 log.cc:824] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:35.067008 11533 log.cc:824] T 65abd25638c44e2ba28089d387a6f88d P 595abd90c2fd4d9c9575eaaf70cd3c19: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:35.067993 11531 tablet_bootstrap.cc:492] T 65abd25638c44e2ba28089d387a6f88d P ba146c9f56cd4a8aa16f6af2125eb2dd: No bootstrap required, opened a new log
I20260928 16:43:35.068122 11532 tablet_bootstrap.cc:492] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085: No bootstrap required, opened a new log
I20260928 16:43:35.068240 11531 ts_tablet_manager.cc:1397] T 65abd25638c44e2ba28089d387a6f88d P ba146c9f56cd4a8aa16f6af2125eb2dd: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 16:43:35.068285 11532 ts_tablet_manager.cc:1397] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:43:35.069123 11533 tablet_bootstrap.cc:492] T 65abd25638c44e2ba28089d387a6f88d P 595abd90c2fd4d9c9575eaaf70cd3c19: No bootstrap required, opened a new log
I20260928 16:43:35.069371 11533 ts_tablet_manager.cc:1397] T 65abd25638c44e2ba28089d387a6f88d P 595abd90c2fd4d9c9575eaaf70cd3c19: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:43:35.072986 11532 raft_consensus.cc:348] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595abd90c2fd4d9c9575eaaf70cd3c19" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41045 } } peers { permanent_uuid: "1944b8253eab45638b870e7314af5085" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43455 } } peers { permanent_uuid: "ba146c9f56cd4a8aa16f6af2125eb2dd" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37097 } }
I20260928 16:43:35.073004 11531 raft_consensus.cc:348] T 65abd25638c44e2ba28089d387a6f88d P ba146c9f56cd4a8aa16f6af2125eb2dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595abd90c2fd4d9c9575eaaf70cd3c19" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41045 } } peers { permanent_uuid: "1944b8253eab45638b870e7314af5085" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43455 } } peers { permanent_uuid: "ba146c9f56cd4a8aa16f6af2125eb2dd" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37097 } }
I20260928 16:43:35.073342 11531 raft_consensus.cc:374] T 65abd25638c44e2ba28089d387a6f88d P ba146c9f56cd4a8aa16f6af2125eb2dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:35.073342 11532 raft_consensus.cc:374] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:35.073395 11531 raft_consensus.cc:729] T 65abd25638c44e2ba28089d387a6f88d P ba146c9f56cd4a8aa16f6af2125eb2dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba146c9f56cd4a8aa16f6af2125eb2dd, State: Initialized, Role: FOLLOWER
I20260928 16:43:35.073406 11532 raft_consensus.cc:729] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1944b8253eab45638b870e7314af5085, State: Initialized, Role: FOLLOWER
I20260928 16:43:35.073398 11533 raft_consensus.cc:348] T 65abd25638c44e2ba28089d387a6f88d P 595abd90c2fd4d9c9575eaaf70cd3c19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595abd90c2fd4d9c9575eaaf70cd3c19" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41045 } } peers { permanent_uuid: "1944b8253eab45638b870e7314af5085" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43455 } } peers { permanent_uuid: "ba146c9f56cd4a8aa16f6af2125eb2dd" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37097 } }
I20260928 16:43:35.073765 11533 raft_consensus.cc:374] T 65abd25638c44e2ba28089d387a6f88d P 595abd90c2fd4d9c9575eaaf70cd3c19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:35.073820 11533 raft_consensus.cc:729] T 65abd25638c44e2ba28089d387a6f88d P 595abd90c2fd4d9c9575eaaf70cd3c19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 595abd90c2fd4d9c9575eaaf70cd3c19, State: Initialized, Role: FOLLOWER
I20260928 16:43:35.074005 11531 consensus_queue.cc:260] T 65abd25638c44e2ba28089d387a6f88d P ba146c9f56cd4a8aa16f6af2125eb2dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595abd90c2fd4d9c9575eaaf70cd3c19" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41045 } } peers { permanent_uuid: "1944b8253eab45638b870e7314af5085" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43455 } } peers { permanent_uuid: "ba146c9f56cd4a8aa16f6af2125eb2dd" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37097 } }
I20260928 16:43:35.074005 11532 consensus_queue.cc:260] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595abd90c2fd4d9c9575eaaf70cd3c19" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41045 } } peers { permanent_uuid: "1944b8253eab45638b870e7314af5085" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43455 } } peers { permanent_uuid: "ba146c9f56cd4a8aa16f6af2125eb2dd" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37097 } }
I20260928 16:43:35.074364 11533 consensus_queue.cc:260] T 65abd25638c44e2ba28089d387a6f88d P 595abd90c2fd4d9c9575eaaf70cd3c19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595abd90c2fd4d9c9575eaaf70cd3c19" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41045 } } peers { permanent_uuid: "1944b8253eab45638b870e7314af5085" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43455 } } peers { permanent_uuid: "ba146c9f56cd4a8aa16f6af2125eb2dd" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37097 } }
I20260928 16:43:35.075163 11514 heartbeater.cc:503] Master 127.25.71.126:35369 was elected leader, sending a full tablet report...
I20260928 16:43:35.075271 11532 ts_tablet_manager.cc:1428] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 16:43:35.075271 11531 ts_tablet_manager.cc:1428] T 65abd25638c44e2ba28089d387a6f88d P ba146c9f56cd4a8aa16f6af2125eb2dd: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 16:43:35.075335 11383 heartbeater.cc:503] Master 127.25.71.126:35369 was elected leader, sending a full tablet report...
I20260928 16:43:35.077162 11252 heartbeater.cc:503] Master 127.25.71.126:35369 was elected leader, sending a full tablet report...
I20260928 16:43:35.077171 11533 ts_tablet_manager.cc:1428] T 65abd25638c44e2ba28089d387a6f88d P 595abd90c2fd4d9c9575eaaf70cd3c19: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260928 16:43:35.114326 11537 raft_consensus.cc:482] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:35.114617 11537 raft_consensus.cc:504] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595abd90c2fd4d9c9575eaaf70cd3c19" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41045 } } peers { permanent_uuid: "1944b8253eab45638b870e7314af5085" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43455 } } peers { permanent_uuid: "ba146c9f56cd4a8aa16f6af2125eb2dd" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37097 } }
I20260928 16:43:35.116228 11537 leader_election.cc:288] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 595abd90c2fd4d9c9575eaaf70cd3c19 (127.25.71.65:41045), ba146c9f56cd4a8aa16f6af2125eb2dd (127.25.71.66:37097)
I20260928 16:43:35.121531 11529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:35.116764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:49213 (local address 127.25.71.66:37097)
0928 16:43:35.116919 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:35.116924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:35.116964 (+ 40us) server_negotiation.cc:407] Connection header received
0928 16:43:35.117030 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:35.117033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:35.117088 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:35.117180 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:35.117738 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.118473 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.120095 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.120705 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.120843 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:35.120902 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:35.121275 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 16:43:35.121375 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 16:43:35.121583 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:35.116975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:53603 (local address 127.25.71.65:41045)
0928 16:43:35.117214 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:35.117219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:35.117302 (+ 83us) server_negotiation.cc:407] Connection header received
0928 16:43:35.117457 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:35.117460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:35.117514 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:35.117632 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:35.118132 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.118886 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.120384 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.120947 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.121087 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:35.121152 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:35.121334 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 16:43:35.121427 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:43:35.121616 11516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:35.116544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:37097 (local address 127.25.71.67:49213)
0928 16:43:35.116722 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:35.116742 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:35.116896 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:35.117283 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:35.117288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:35.117351 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:35.117521 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:35.117533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.118619 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.118653 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:35.119857 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:35.119872 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.120943 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.120947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:35.121122 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:35.121150 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 16:43:35.121237 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 16:43:35.121345 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":1,"spinlock_wait_cycles":15616}
I20260928 16:43:35.121656 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:35.116544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:41045 (local address 127.25.71.67:53603)
0928 16:43:35.117197 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:35.117211 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:35.117359 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:35.117738 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:35.117741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:35.117764 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:35.117931 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:35.117938 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.119068 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.119072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:35.120219 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:35.120242 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.121081 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.121084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:35.121182 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:35.121210 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 16:43:35.121302 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 16:43:35.121397 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":134,"spinlock_wait_cycles":22144,"thread_start_us":159,"threads_started":1}
I20260928 16:43:35.122593 11206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65abd25638c44e2ba28089d387a6f88d" candidate_uuid: "1944b8253eab45638b870e7314af5085" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "595abd90c2fd4d9c9575eaaf70cd3c19" is_pre_election: true
I20260928 16:43:35.122597 11337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65abd25638c44e2ba28089d387a6f88d" candidate_uuid: "1944b8253eab45638b870e7314af5085" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba146c9f56cd4a8aa16f6af2125eb2dd" is_pre_election: true
I20260928 16:43:35.123080 11206 raft_consensus.cc:2454] T 65abd25638c44e2ba28089d387a6f88d P 595abd90c2fd4d9c9575eaaf70cd3c19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1944b8253eab45638b870e7314af5085 in term 0.
I20260928 16:43:35.123080 11337 raft_consensus.cc:2454] T 65abd25638c44e2ba28089d387a6f88d P ba146c9f56cd4a8aa16f6af2125eb2dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1944b8253eab45638b870e7314af5085 in term 0.
I20260928 16:43:35.123739 11402 leader_election.cc:302] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1944b8253eab45638b870e7314af5085, 595abd90c2fd4d9c9575eaaf70cd3c19, ba146c9f56cd4a8aa16f6af2125eb2dd; no voters:
I20260928 16:43:35.124092 11537 raft_consensus.cc:2781] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:43:35.124222 11537 raft_consensus.cc:482] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:43:35.124270 11537 raft_consensus.cc:3037] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:35.125810 11537 raft_consensus.cc:504] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595abd90c2fd4d9c9575eaaf70cd3c19" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41045 } } peers { permanent_uuid: "1944b8253eab45638b870e7314af5085" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43455 } } peers { permanent_uuid: "ba146c9f56cd4a8aa16f6af2125eb2dd" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37097 } }
I20260928 16:43:35.126323 11537 leader_election.cc:288] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 [CANDIDATE]: Term 1 election: Requested vote from peers 595abd90c2fd4d9c9575eaaf70cd3c19 (127.25.71.65:41045), ba146c9f56cd4a8aa16f6af2125eb2dd (127.25.71.66:37097)
I20260928 16:43:35.126864 11337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65abd25638c44e2ba28089d387a6f88d" candidate_uuid: "1944b8253eab45638b870e7314af5085" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba146c9f56cd4a8aa16f6af2125eb2dd"
I20260928 16:43:35.126910 11206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65abd25638c44e2ba28089d387a6f88d" candidate_uuid: "1944b8253eab45638b870e7314af5085" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "595abd90c2fd4d9c9575eaaf70cd3c19"
I20260928 16:43:35.127034 11337 raft_consensus.cc:3037] T 65abd25638c44e2ba28089d387a6f88d P ba146c9f56cd4a8aa16f6af2125eb2dd [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:35.127113 11206 raft_consensus.cc:3037] T 65abd25638c44e2ba28089d387a6f88d P 595abd90c2fd4d9c9575eaaf70cd3c19 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:35.128389 11337 raft_consensus.cc:2454] T 65abd25638c44e2ba28089d387a6f88d P ba146c9f56cd4a8aa16f6af2125eb2dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1944b8253eab45638b870e7314af5085 in term 1.
I20260928 16:43:35.128515 11206 raft_consensus.cc:2454] T 65abd25638c44e2ba28089d387a6f88d P 595abd90c2fd4d9c9575eaaf70cd3c19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1944b8253eab45638b870e7314af5085 in term 1.
I20260928 16:43:35.128841 11403 leader_election.cc:302] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1944b8253eab45638b870e7314af5085, ba146c9f56cd4a8aa16f6af2125eb2dd; no voters:
I20260928 16:43:35.129076 11537 raft_consensus.cc:2781] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:35.129346 11537 raft_consensus.cc:686] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 [term 1 LEADER]: Becoming Leader. State: Replica: 1944b8253eab45638b870e7314af5085, State: Running, Role: LEADER
I20260928 16:43:35.129745 11537 consensus_queue.cc:237] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595abd90c2fd4d9c9575eaaf70cd3c19" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41045 } } peers { permanent_uuid: "1944b8253eab45638b870e7314af5085" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43455 } } peers { permanent_uuid: "ba146c9f56cd4a8aa16f6af2125eb2dd" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37097 } }
I20260928 16:43:35.133504 11062 catalog_manager.cc:5495] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1944b8253eab45638b870e7314af5085 (127.25.71.67). New cstate: current_term: 1 leader_uuid: "1944b8253eab45638b870e7314af5085" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595abd90c2fd4d9c9575eaaf70cd3c19" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1944b8253eab45638b870e7314af5085" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba146c9f56cd4a8aa16f6af2125eb2dd" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37097 } health_report { overall_health: UNKNOWN } } }
I20260928 16:43:35.152786 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:35.147701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52224 (local address 127.25.71.67:43455)
0928 16:43:35.147952 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:35.147958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:35.147976 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:43:35.148093 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:35.148095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:35.148172 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:35.148300 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:35.148763 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.149473 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.150698 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.150972 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.151092 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:35.151523 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:35.152223 (+ 700us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:35.152347 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:35.152466 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 16:43:35.152581 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 16:43:35.160683 11206 raft_consensus.cc:1261] T 65abd25638c44e2ba28089d387a6f88d P 595abd90c2fd4d9c9575eaaf70cd3c19 [term 1 FOLLOWER]: Refusing update from remote peer 1944b8253eab45638b870e7314af5085: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:35.160800 11337 raft_consensus.cc:1261] T 65abd25638c44e2ba28089d387a6f88d P ba146c9f56cd4a8aa16f6af2125eb2dd [term 1 FOLLOWER]: Refusing update from remote peer 1944b8253eab45638b870e7314af5085: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:35.161615 11541 consensus_queue.cc:1035] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 [LEADER]: Connected to new peer: Peer: permanent_uuid: "595abd90c2fd4d9c9575eaaf70cd3c19" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 41045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:35.161947 11537 consensus_queue.cc:1035] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba146c9f56cd4a8aa16f6af2125eb2dd" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:35.171717 11546 mvcc.cc:205] Tried to move back new op lower bound from 7334354186889433088 to 7334354186774552576. Current Snapshot: MvccSnapshot[applied={T|T < 7334354186889433088}]
I20260928 16:43:35.174279 11545 mvcc.cc:205] Tried to move back new op lower bound from 7334354186889433088 to 7334354186774552576. Current Snapshot: MvccSnapshot[applied={T|T < 7334354186889433088}]
I20260928 16:43:35.181934 11062 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:35.182171 11062 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:35.185452 11062 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 241c3e1cff20455790169b17e2c0ff33: Sending DeleteTablet for 3 replicas of tablet 65abd25638c44e2ba28089d387a6f88d
I20260928 16:43:35.186717 11186 tablet_service.cc:1514] Processing DeleteTablet for tablet 65abd25638c44e2ba28089d387a6f88d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:35 UTC) from {username='slave'} at 127.0.0.1:47550
I20260928 16:43:35.186779 11448 tablet_service.cc:1514] Processing DeleteTablet for tablet 65abd25638c44e2ba28089d387a6f88d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:35 UTC) from {username='slave'} at 127.0.0.1:52212
I20260928 16:43:35.187427 11317 tablet_service.cc:1514] Processing DeleteTablet for tablet 65abd25638c44e2ba28089d387a6f88d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:35 UTC) from {username='slave'} at 127.0.0.1:33466
I20260928 16:43:35.187518 11555 tablet_replica.cc:330] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085: stopping tablet replica
I20260928 16:43:35.187572 11554 tablet_replica.cc:330] T 65abd25638c44e2ba28089d387a6f88d P 595abd90c2fd4d9c9575eaaf70cd3c19: stopping tablet replica
I20260928 16:43:35.188027 11554 raft_consensus.cc:2229] T 65abd25638c44e2ba28089d387a6f88d P 595abd90c2fd4d9c9575eaaf70cd3c19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:35.188027 11555 raft_consensus.cc:2229] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:43:35.188242 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 11124
I20260928 16:43:35.188472 11554 raft_consensus.cc:2258] T 65abd25638c44e2ba28089d387a6f88d P 595abd90c2fd4d9c9575eaaf70cd3c19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:35.189582 11556 tablet_replica.cc:330] T 65abd25638c44e2ba28089d387a6f88d P ba146c9f56cd4a8aa16f6af2125eb2dd: stopping tablet replica
I20260928 16:43:35.189770 11555 raft_consensus.cc:2258] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:35.190027 11556 raft_consensus.cc:2229] T 65abd25638c44e2ba28089d387a6f88d P ba146c9f56cd4a8aa16f6af2125eb2dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:35.190405 11556 raft_consensus.cc:2258] T 65abd25638c44e2ba28089d387a6f88d P ba146c9f56cd4a8aa16f6af2125eb2dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:35.192031 11556 ts_tablet_manager.cc:1905] T 65abd25638c44e2ba28089d387a6f88d P ba146c9f56cd4a8aa16f6af2125eb2dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:35.192031 11555 ts_tablet_manager.cc:1905] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:35.195454 11555 ts_tablet_manager.cc:1918] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:43:35.195580 11556 ts_tablet_manager.cc:1918] T 65abd25638c44e2ba28089d387a6f88d P ba146c9f56cd4a8aa16f6af2125eb2dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:43:35.195686 11556 log.cc:1196] T 65abd25638c44e2ba28089d387a6f88d P ba146c9f56cd4a8aa16f6af2125eb2dd: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/65abd25638c44e2ba28089d387a6f88d
I20260928 16:43:35.195832 11555 log.cc:1196] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/65abd25638c44e2ba28089d387a6f88d
I20260928 16:43:35.196044 11556 ts_tablet_manager.cc:1939] T 65abd25638c44e2ba28089d387a6f88d P ba146c9f56cd4a8aa16f6af2125eb2dd: Deleting consensus metadata
I20260928 16:43:35.196316 11555 ts_tablet_manager.cc:1939] T 65abd25638c44e2ba28089d387a6f88d P 1944b8253eab45638b870e7314af5085: Deleting consensus metadata
I20260928 16:43:35.197870 11051 catalog_manager.cc:4841] TS 1944b8253eab45638b870e7314af5085 (127.25.71.67:43455): tablet 65abd25638c44e2ba28089d387a6f88d (table test-table [id=310ee05de97a4674880cc043005ef86d]) successfully deleted
I20260928 16:43:35.197870 11050 catalog_manager.cc:4841] TS ba146c9f56cd4a8aa16f6af2125eb2dd (127.25.71.66:37097): tablet 65abd25638c44e2ba28089d387a6f88d (table test-table [id=310ee05de97a4674880cc043005ef86d]) successfully deleted
W20260928 16:43:35.199076 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.71.65:41045 (error 108)
I20260928 16:43:35.199366 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 11255
I20260928 16:43:35.201066 11527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:35.200631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:41045 (local address 127.0.0.1:47552)
0928 16:43:35.200823 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:35.200945 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.65:41045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 16:43:35.201524 11049 catalog_manager.cc:4584] TS 595abd90c2fd4d9c9575eaaf70cd3c19 (127.25.71.65:41045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 65abd25638c44e2ba28089d387a6f88d: Network error: Client connection negotiation failed: client connection to 127.25.71.65:41045: connect: Connection refused (error 111)
I20260928 16:43:35.206017 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 11386
I20260928 16:43:35.212213 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 11033
2026-09-28T16:43:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1186 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:35 dist-test-slave-nqgh krb5kdc[11560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:35 dist-test-slave-nqgh krb5kdc[11560](info): set up 2 sockets
Sep 28 16:43:35 dist-test-slave-nqgh krb5kdc[11560](info): commencing operation
krb5kdc: starting...
W20260928 16:43:36.188596 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:43:36 dist-test-slave-nqgh krb5kdc[11560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:43:36.331212 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:37147
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:42309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:37147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:36.468309 11573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:36.468708 11573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:36.468775 11573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:36.473385 11573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:36.473497 11573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:36.473528 11573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:36.473551 11573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:36.473572 11573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:36.480134 11573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:42309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:37147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:37147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:36.481796 11573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:36.483253 11573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:36.494490 11579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:36.494478 11581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:36.494478 11578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:36.495357 11573 server_base.cc:1034] running on GCE node
I20260928 16:43:36.496239 11573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:36.497850 11573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:36.499035 11573 hybrid_clock.cc:648] HybridClock initialized: now 1790613816499021 us; error 48 us; skew 500 ppm
Sep 28 16:43:36 dist-test-slave-nqgh krb5kdc[11560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613816, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:36.511540 11573 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:43:36.512988 11573 webserver.cc:460] Webserver started at http://127.25.71.126:34239/ using document root <none> and password file <none>
I20260928 16:43:36.513759 11573 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:36.513849 11573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:36.514117 11573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:36.516939 11573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "2129bb0743fd46d4b3a973864dd2dc9c"
format_stamp: "Formatted at 2026-09-28 16:43:36 on dist-test-slave-nqgh"
server_key: "524d137424ccd09a6296d21bfa50b92c"
server_key_iv: "d4dc275856d92752959939e4928448f1"
server_key_version: "encryptionkey@0"
I20260928 16:43:36.517630 11573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "2129bb0743fd46d4b3a973864dd2dc9c"
format_stamp: "Formatted at 2026-09-28 16:43:36 on dist-test-slave-nqgh"
server_key: "524d137424ccd09a6296d21bfa50b92c"
server_key_iv: "d4dc275856d92752959939e4928448f1"
server_key_version: "encryptionkey@0"
I20260928 16:43:36.522843 11573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 16:43:36.526255 11588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:36.528123 11573 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:43:36.528337 11573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "2129bb0743fd46d4b3a973864dd2dc9c"
format_stamp: "Formatted at 2026-09-28 16:43:36 on dist-test-slave-nqgh"
server_key: "524d137424ccd09a6296d21bfa50b92c"
server_key_iv: "d4dc275856d92752959939e4928448f1"
server_key_version: "encryptionkey@0"
I20260928 16:43:36.528505 11573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:36.544857 11573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:36.548828 11573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:36.549062 11573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:36.558247 11573 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:37147
I20260928 16:43:36.558319 11640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:37147 every 8 connection(s)
I20260928 16:43:36.559573 11573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:36.560485 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 11573
I20260928 16:43:36.560644 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:43:36.560983 25885 external_mini_cluster.cc:1428] Setting key 7867395e0ee6fab048bcf831d07a9306
I20260928 16:43:36.563503 11641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:36.569849 11641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c: Bootstrap starting.
Sep 28 16:43:36 dist-test-slave-nqgh krb5kdc[11560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:43:36.572584 11641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:36.573396 11641 log.cc:824] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:36.576014 11641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c: No bootstrap required, opened a new log
I20260928 16:43:36.578866 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:36.562868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35942 (local address 127.25.71.126:37147)
0928 16:43:36.563477 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:36.563498 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:36.563553 (+ 55us) server_negotiation.cc:407] Connection header received
0928 16:43:36.564505 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:36.564531 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:36.564987 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:36.565393 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:36.566376 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:36.567623 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:36.568636 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:36.569261 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:36.569440 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:36.572855 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:36.572862 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:36.572885 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:36.572935 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:36.575219 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:36.575773 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:36.575779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:36.575787 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:36.575932 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:36.576246 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:36.576249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:36.576250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:36.576807 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:36.577034 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:36.577373 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 16:43:36.577793 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":196,"threads_started":1}
I20260928 16:43:36.580122 11641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2129bb0743fd46d4b3a973864dd2dc9c" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37147 } }
I20260928 16:43:36.580469 11641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:36.580545 11641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2129bb0743fd46d4b3a973864dd2dc9c, State: Initialized, Role: FOLLOWER
I20260928 16:43:36.581063 11641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2129bb0743fd46d4b3a973864dd2dc9c" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37147 } }
I20260928 16:43:36.581226 11641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:36.581295 11641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:36.581390 11641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:36.582492 11641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2129bb0743fd46d4b3a973864dd2dc9c" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37147 } }
I20260928 16:43:36.582911 11641 leader_election.cc:302] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2129bb0743fd46d4b3a973864dd2dc9c; no voters:
I20260928 16:43:36.583245 11641 leader_election.cc:288] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:36.583536 11646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:36.583920 11646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c [term 1 LEADER]: Becoming Leader. State: Replica: 2129bb0743fd46d4b3a973864dd2dc9c, State: Running, Role: LEADER
I20260928 16:43:36.584343 11646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2129bb0743fd46d4b3a973864dd2dc9c" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37147 } }
I20260928 16:43:36.584578 11641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:36.586598 11648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2129bb0743fd46d4b3a973864dd2dc9c. Latest consensus state: current_term: 1 leader_uuid: "2129bb0743fd46d4b3a973864dd2dc9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2129bb0743fd46d4b3a973864dd2dc9c" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37147 } } }
I20260928 16:43:36.586839 11648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:36.586598 11647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2129bb0743fd46d4b3a973864dd2dc9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2129bb0743fd46d4b3a973864dd2dc9c" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37147 } } }
I20260928 16:43:36.588941 11647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c [sys.catalog]: This master's current role is: LEADER
W20260928 16:43:36.590934 11660 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:43:36.591060 11660 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:43:36.591351 11661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:36.592224 11661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:36.599833 11661 catalog_manager.cc:1347] Generated new cluster ID: 8668dbe3aba043318850fa889192a012
I20260928 16:43:36.599956 11661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:36.618134 11661 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:36.619481 11661 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:36.627604 11661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c: Generated new TSK 0
I20260928 16:43:36.628743 11661 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
I20260928 16:43:36.666875 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:37147
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:42309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:36.804011 11667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:36.804409 11667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:36.804481 11667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:36.809224 11667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:36.809337 11667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:36.809432 11667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:43:36.816038 11667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:42309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:37147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:36.817808 11667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:36.819329 11667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:36.832121 11675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:36.832219 11672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:36.832163 11673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:36.832509 11667 server_base.cc:1034] running on GCE node
I20260928 16:43:36.833288 11667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:36.834334 11667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:36.835554 11667 hybrid_clock.cc:648] HybridClock initialized: now 1790613816835526 us; error 63 us; skew 500 ppm
Sep 28 16:43:36 dist-test-slave-nqgh krb5kdc[11560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613816, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:36.838845 11667 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:43:36.839903 11667 webserver.cc:460] Webserver started at http://127.25.71.65:36661/ using document root <none> and password file <none>
I20260928 16:43:36.840587 11667 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:36.840662 11667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:36.840855 11667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:36.843271 11667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "141d428d501f4a12a7458e788e1ed9f1"
format_stamp: "Formatted at 2026-09-28 16:43:36 on dist-test-slave-nqgh"
server_key: "97e9be5c9e5ecb944315fcd6f35f1cfe"
server_key_iv: "a0d4d246690253d9f986a145df1be822"
server_key_version: "encryptionkey@0"
I20260928 16:43:36.843878 11667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "141d428d501f4a12a7458e788e1ed9f1"
format_stamp: "Formatted at 2026-09-28 16:43:36 on dist-test-slave-nqgh"
server_key: "97e9be5c9e5ecb944315fcd6f35f1cfe"
server_key_iv: "a0d4d246690253d9f986a145df1be822"
server_key_version: "encryptionkey@0"
I20260928 16:43:36.848861 11667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:43:36.852203 11682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:36.854552 11667 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:43:36.854785 11667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "141d428d501f4a12a7458e788e1ed9f1"
format_stamp: "Formatted at 2026-09-28 16:43:36 on dist-test-slave-nqgh"
server_key: "97e9be5c9e5ecb944315fcd6f35f1cfe"
server_key_iv: "a0d4d246690253d9f986a145df1be822"
server_key_version: "encryptionkey@0"
I20260928 16:43:36.855007 11667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:36.866516 11667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:36.870729 11667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:36.870944 11667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:36.871618 11667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:36.872570 11667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:36.872624 11667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:36.872665 11667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:36.872697 11667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:36.882947 11667 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:33623
I20260928 16:43:36.883033 11795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:33623 every 8 connection(s)
I20260928 16:43:36.884053 11667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:43:36.886052 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 11667
I20260928 16:43:36.886217 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:43:36.886516 25885 external_mini_cluster.cc:1428] Setting key bdc39476b474e1be693fd6fcd97536d4
WARNING: no policy specified for kudu/127.25.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.66@KRBTEST.COM" created.
Sep 28 16:43:36 dist-test-slave-nqgh krb5kdc[11560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613816, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:43:36.905428 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:36.886850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:52875 (local address 127.25.71.126:37147)
0928 16:43:36.887057 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:36.887063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:36.888395 (+ 1332us) server_negotiation.cc:407] Connection header received
0928 16:43:36.890086 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:36.890090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:36.890167 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:36.890246 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:36.891922 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:36.892535 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:36.893831 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:36.894090 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:36.894192 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:36.900632 (+ 6440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:36.900638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:36.900642 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:36.900688 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:36.902713 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:36.903484 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:36.903488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:36.903490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:36.903588 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:36.903994 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:36.903997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:36.903999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:36.904265 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:36.904360 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:36.905028 (+ 668us) server_negotiation.cc:299] Negotiation successful
0928 16:43:36.905221 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:43:36.906548 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:36.887489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37147 (local address 127.25.71.65:52875)
0928 16:43:36.888172 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:36.888240 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:36.889796 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:36.890505 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:36.890514 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:36.891092 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:36.891742 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:36.891760 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:36.892671 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:36.892675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:36.893701 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:36.893712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:36.894270 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:36.894275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:36.894499 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:36.895250 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:36.895271 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:36.900307 (+ 5036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:36.902911 (+ 2604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:36.902920 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:36.902942 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:36.903353 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:36.903678 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:36.903681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:36.903684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:36.903883 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:36.904373 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:36.904383 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:36.904741 (+ 358us) client_negotiation.cc:769] Sending connection context
0928 16:43:36.904963 (+ 222us) client_negotiation.cc:240] Negotiation successful
0928 16:43:36.905421 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":205,"threads_started":1}
I20260928 16:43:36.908394 11796 heartbeater.cc:347] Connected to a master server at 127.25.71.126:37147
I20260928 16:43:36.908730 11796 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:36.909406 11796 heartbeater.cc:511] Master 127.25.71.126:37147 requested a full tablet report, sending...
I20260928 16:43:36.911612 11605 ts_manager.cc:194] Registered new tserver with Master: 141d428d501f4a12a7458e788e1ed9f1 (127.25.71.65:33623)
I20260928 16:43:36.912920 11605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:52875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
I20260928 16:43:36.919353 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:37147
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:42309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:37.053716 11801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:37.054131 11801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:37.054203 11801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:37.059224 11801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:37.059334 11801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:37.059427 11801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:43:37.066100 11801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:42309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:37147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:37.067746 11801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:37.069190 11801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:37.082499 11809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:37.082505 11806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:37.082870 11801 server_base.cc:1034] running on GCE node
W20260928 16:43:37.082471 11807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:37.083611 11801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:37.084582 11801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:37.085886 11801 hybrid_clock.cc:648] HybridClock initialized: now 1790613817085814 us; error 97 us; skew 500 ppm
Sep 28 16:43:37 dist-test-slave-nqgh krb5kdc[11560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613817, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:37.089757 11801 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:43:37.091149 11801 webserver.cc:460] Webserver started at http://127.25.71.66:34933/ using document root <none> and password file <none>
I20260928 16:43:37.091920 11801 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:37.092001 11801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:37.092228 11801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:37.095016 11801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "8338b65caaf949c2bac7329fce4db370"
format_stamp: "Formatted at 2026-09-28 16:43:37 on dist-test-slave-nqgh"
server_key: "69bea05386a46bf7b32023d45820a913"
server_key_iv: "30b6d4225b3e92feb403516002dd216e"
server_key_version: "encryptionkey@0"
I20260928 16:43:37.095748 11801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "8338b65caaf949c2bac7329fce4db370"
format_stamp: "Formatted at 2026-09-28 16:43:37 on dist-test-slave-nqgh"
server_key: "69bea05386a46bf7b32023d45820a913"
server_key_iv: "30b6d4225b3e92feb403516002dd216e"
server_key_version: "encryptionkey@0"
I20260928 16:43:37.100948 11801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:43:37.104604 11816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:37.106606 11801 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 16:43:37.106881 11801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "8338b65caaf949c2bac7329fce4db370"
format_stamp: "Formatted at 2026-09-28 16:43:37 on dist-test-slave-nqgh"
server_key: "69bea05386a46bf7b32023d45820a913"
server_key_iv: "30b6d4225b3e92feb403516002dd216e"
server_key_version: "encryptionkey@0"
I20260928 16:43:37.107043 11801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:37.125650 11801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:37.130553 11801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:37.130851 11801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:37.131690 11801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:37.132822 11801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:37.132882 11801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:37.132934 11801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:37.132957 11801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:37.144984 11801 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:37423
I20260928 16:43:37.145186 11930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:37423 every 8 connection(s)
I20260928 16:43:37.146302 11801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:43:37.148455 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 11801
I20260928 16:43:37.148620 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:43:37.148941 25885 external_mini_cluster.cc:1428] Setting key 43948a79ac8e41dd990a09fe720a8339
WARNING: no policy specified for kudu/127.25.71.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.67@KRBTEST.COM" created.
Sep 28 16:43:37 dist-test-slave-nqgh krb5kdc[11560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613817, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:43:37.171067 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:37.149018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:59799 (local address 127.25.71.126:37147)
0928 16:43:37.149209 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:37.149214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:37.150897 (+ 1683us) server_negotiation.cc:407] Connection header received
0928 16:43:37.153522 (+ 2625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:37.153526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:37.153722 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:37.153973 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:37.156096 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.156738 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.158234 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.158499 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.158608 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:37.165961 (+ 7353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:37.165965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:37.165968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:37.166010 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:37.167873 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:37.168818 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:37.168822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:37.168823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:37.168939 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:37.169390 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:37.169393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:37.169394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:37.169676 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:37.169794 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:37.170575 (+ 781us) server_negotiation.cc:299] Negotiation successful
0928 16:43:37.170830 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 16:43:37.172422 11933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:37.149722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37147 (local address 127.25.71.66:59799)
0928 16:43:37.150618 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:37.150756 (+ 138us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:37.153012 (+ 2256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:37.154389 (+ 1377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:37.154405 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:37.155086 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:37.155839 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:37.155861 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.156908 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.156911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:37.158045 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:37.158063 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.158842 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.158857 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:37.159129 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:37.160217 (+ 1088us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:37.160251 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:37.165617 (+ 5366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:37.168174 (+ 2557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:37.168182 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:37.168206 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:37.168654 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:37.169052 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:37.169056 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:37.169058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:37.169265 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:37.169805 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:37.169816 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:37.170228 (+ 412us) client_negotiation.cc:769] Sending connection context
0928 16:43:37.170472 (+ 244us) client_negotiation.cc:240] Negotiation successful
0928 16:43:37.171035 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":258,"threads_started":1}
I20260928 16:43:37.174409 11931 heartbeater.cc:347] Connected to a master server at 127.25.71.126:37147
I20260928 16:43:37.174849 11931 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:37.175576 11931 heartbeater.cc:511] Master 127.25.71.126:37147 requested a full tablet report, sending...
I20260928 16:43:37.177193 11605 ts_manager.cc:194] Registered new tserver with Master: 8338b65caaf949c2bac7329fce4db370 (127.25.71.66:37423)
I20260928 16:43:37.178025 11605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:59799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
I20260928 16:43:37.183565 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:37147
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:42309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:37.310426 11936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:37.310762 11936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:37.310808 11936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:37.315315 11936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:37.315407 11936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:37.315474 11936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:43:37.321396 11936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:42309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:37147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:37.322595 11936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:37.323724 11936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:37.335655 11941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:37.335767 11942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:37.335656 11944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:37.336076 11936 server_base.cc:1034] running on GCE node
I20260928 16:43:37.336619 11936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:37.337337 11936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:37.338542 11936 hybrid_clock.cc:648] HybridClock initialized: now 1790613817338509 us; error 53 us; skew 500 ppm
Sep 28 16:43:37 dist-test-slave-nqgh krb5kdc[11560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613817, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:37.342126 11936 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:43:37.343456 11936 webserver.cc:460] Webserver started at http://127.25.71.67:42561/ using document root <none> and password file <none>
I20260928 16:43:37.344177 11936 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:37.344251 11936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:37.344508 11936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:37.347245 11936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "496139b1aa904b1eb6a9f6cefaa16ced"
format_stamp: "Formatted at 2026-09-28 16:43:37 on dist-test-slave-nqgh"
server_key: "d4dd483314c0e68d020fc50821460196"
server_key_iv: "891c93ffb97e25bdc5181f17c5f95012"
server_key_version: "encryptionkey@0"
I20260928 16:43:37.347929 11936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "496139b1aa904b1eb6a9f6cefaa16ced"
format_stamp: "Formatted at 2026-09-28 16:43:37 on dist-test-slave-nqgh"
server_key: "d4dd483314c0e68d020fc50821460196"
server_key_iv: "891c93ffb97e25bdc5181f17c5f95012"
server_key_version: "encryptionkey@0"
I20260928 16:43:37.353462 11936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:43:37.357316 11951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:37.359231 11936 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:43:37.359414 11936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "496139b1aa904b1eb6a9f6cefaa16ced"
format_stamp: "Formatted at 2026-09-28 16:43:37 on dist-test-slave-nqgh"
server_key: "d4dd483314c0e68d020fc50821460196"
server_key_iv: "891c93ffb97e25bdc5181f17c5f95012"
server_key_version: "encryptionkey@0"
I20260928 16:43:37.359614 11936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:37.376585 11936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:37.381244 11936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:37.381474 11936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:37.382318 11936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:37.383565 11936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:37.383630 11936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:37.383694 11936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:37.383723 11936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:37.395401 11936 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:36623
I20260928 16:43:37.395522 12064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:36623 every 8 connection(s)
I20260928 16:43:37.396653 11936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:43:37.402402 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 11936
I20260928 16:43:37.402544 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:43:37.402971 25885 external_mini_cluster.cc:1428] Setting key fef762193eeacca72825ef220b6c2bbc
Sep 28 16:43:37 dist-test-slave-nqgh krb5kdc[11560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613817, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:43:37.415583 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:37.399062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:48561 (local address 127.25.71.126:37147)
0928 16:43:37.399285 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:37.399290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:37.400403 (+ 1113us) server_negotiation.cc:407] Connection header received
0928 16:43:37.402076 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:37.402080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:37.402171 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:37.402256 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:37.404550 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.405351 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.407285 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.407623 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.407762 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:37.411242 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:37.411247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:37.411249 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:37.411288 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:37.412914 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:37.413687 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:37.413689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:37.413691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:37.413779 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:37.414148 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:37.414151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:37.414153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:37.414388 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:37.414480 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:37.415210 (+ 730us) server_negotiation.cc:299] Negotiation successful
0928 16:43:37.415382 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 16:43:37.416782 12067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:37.399400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:37147 (local address 127.25.71.67:48561)
0928 16:43:37.400167 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:37.400249 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:37.401771 (+ 1522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:37.402537 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:37.402550 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:37.403378 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:37.404346 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:37.404376 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.405581 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.405586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:37.406923 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:37.406943 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.407832 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.407835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:37.408085 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:37.408926 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:37.408945 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:37.410974 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:37.413133 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:37.413143 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:37.413172 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:37.413553 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:37.413866 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:37.413869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:37.413872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:37.414047 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:37.414492 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:37.414502 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:37.414926 (+ 424us) client_negotiation.cc:769] Sending connection context
0928 16:43:37.415144 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 16:43:37.415594 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":202,"threads_started":1}
I20260928 16:43:37.418365 12065 heartbeater.cc:347] Connected to a master server at 127.25.71.126:37147
I20260928 16:43:37.418800 12065 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:37.419452 12065 heartbeater.cc:511] Master 127.25.71.126:37147 requested a full tablet report, sending...
I20260928 16:43:37.420972 11605 ts_manager.cc:194] Registered new tserver with Master: 496139b1aa904b1eb6a9f6cefaa16ced (127.25.71.67:36623)
I20260928 16:43:37.421835 11605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:48561
I20260928 16:43:37.431978 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:43:37.445151 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:37.434472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35954 (local address 127.25.71.126:37147)
0928 16:43:37.434667 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:37.434673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:37.435041 (+ 368us) server_negotiation.cc:407] Connection header received
0928 16:43:37.435214 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:37.435217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:37.435281 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:37.435354 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:37.436260 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.436891 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.437944 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.438310 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.438450 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:37.440583 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:37.440587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:37.440590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:37.440637 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:37.442305 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:37.443200 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:37.443204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:37.443205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:37.443330 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:37.443800 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:37.443803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:37.443804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:37.444216 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:37.444371 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:37.444757 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 16:43:37.444880 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:43:37.449700 11605 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:43:37.453884 11605 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:43:37.473491 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:37.467148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49400 (local address 127.25.71.65:33623)
0928 16:43:37.467581 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:37.467589 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:37.468134 (+ 545us) server_negotiation.cc:407] Connection header received
0928 16:43:37.468245 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:37.468251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:37.468533 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:37.468704 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:37.469132 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.470286 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.471776 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.472596 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.472707 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:37.472913 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:37.473179 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 16:43:37.473299 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":84,"threads_started":1}
I20260928 16:43:37.473482 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:37.467034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:33623 (local address 127.0.0.1:49400)
0928 16:43:37.467685 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:37.467751 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:37.467933 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:37.468733 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:37.468740 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:37.468769 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:37.468971 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:37.468981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.470394 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.470397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:37.471656 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:37.471665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.472723 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.472726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:37.472976 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:37.473018 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 16:43:37.473129 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 16:43:37.473293 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":77,"thread_start_us":116,"threads_started":1}
I20260928 16:43:37.474916 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:37.468632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57386 (local address 127.25.71.67:36623)
0928 16:43:37.469098 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:37.469111 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:37.469137 (+ 26us) server_negotiation.cc:407] Connection header received
0928 16:43:37.469234 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:37.469239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:37.469455 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:37.469633 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:37.469900 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.471210 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.473308 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.474190 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.474314 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:37.474476 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:37.474557 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 16:43:37.474720 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":161,"threads_started":1}
I20260928 16:43:37.475077 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:37.467968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:37423 (local address 127.0.0.1:35178)
0928 16:43:37.469385 (+ 1417us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:37.469400 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:37.469518 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:37.469955 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:37.469958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:37.469974 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:37.470054 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:37.470059 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.471430 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.471433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:37.472931 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:37.472979 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.474593 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.474597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:37.474730 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:37.474757 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:43:37.474830 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 16:43:37.474929 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1349,"mutex_wait_us":32,"thread_start_us":400,"threads_started":1}
I20260928 16:43:37.475500 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:37.468252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35178 (local address 127.25.71.66:37423)
0928 16:43:37.469229 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:37.469243 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:37.469526 (+ 283us) server_negotiation.cc:407] Connection header received
0928 16:43:37.469625 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:37.469630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:37.469800 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:37.469960 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:37.470151 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.471280 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.473152 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.474190 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.474283 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:37.474476 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:37.475257 (+ 781us) server_negotiation.cc:299] Negotiation successful
0928 16:43:37.475379 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":180,"threads_started":1}
I20260928 16:43:37.475940 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:37.467173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:36623 (local address 127.0.0.1:57386)
0928 16:43:37.467738 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:37.467755 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:37.467972 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:37.469647 (+ 1675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:37.469650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:37.469664 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:37.469779 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:37.469784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.471417 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.471421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:37.473174 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:37.473183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.474330 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.474334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:37.474429 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:37.474453 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:43:37.474515 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 16:43:37.474598 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":39,"thread_start_us":162,"threads_started":1}
I20260928 16:43:37.477650 11730 tablet_service.cc:1467] Processing CreateTablet for tablet 5254d4c43bb346809a3a212e691f7d9c (DEFAULT_TABLE table=test-table [id=6d270f2fe3e34f47b341d52b8dc13462]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:37.479048 11730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5254d4c43bb346809a3a212e691f7d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:37.478897 11865 tablet_service.cc:1467] Processing CreateTablet for tablet 5254d4c43bb346809a3a212e691f7d9c (DEFAULT_TABLE table=test-table [id=6d270f2fe3e34f47b341d52b8dc13462]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:37.478897 11999 tablet_service.cc:1467] Processing CreateTablet for tablet 5254d4c43bb346809a3a212e691f7d9c (DEFAULT_TABLE table=test-table [id=6d270f2fe3e34f47b341d52b8dc13462]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:37.480170 11865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5254d4c43bb346809a3a212e691f7d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:37.480175 11999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5254d4c43bb346809a3a212e691f7d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:37.488760 12082 tablet_bootstrap.cc:492] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1: Bootstrap starting.
I20260928 16:43:37.489748 12084 tablet_bootstrap.cc:492] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370: Bootstrap starting.
I20260928 16:43:37.491030 12083 tablet_bootstrap.cc:492] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced: Bootstrap starting.
I20260928 16:43:37.492368 12082 tablet_bootstrap.cc:654] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:37.492719 12084 tablet_bootstrap.cc:654] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:37.493268 12083 tablet_bootstrap.cc:654] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:37.493620 12082 log.cc:824] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:37.493739 12084 log.cc:824] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:37.494477 12083 log.cc:824] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:37.496395 12082 tablet_bootstrap.cc:492] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1: No bootstrap required, opened a new log
I20260928 16:43:37.496613 12083 tablet_bootstrap.cc:492] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced: No bootstrap required, opened a new log
I20260928 16:43:37.496680 12082 ts_tablet_manager.cc:1397] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 16:43:37.496856 12083 ts_tablet_manager.cc:1397] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:43:37.497925 12084 tablet_bootstrap.cc:492] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370: No bootstrap required, opened a new log
I20260928 16:43:37.498134 12084 ts_tablet_manager.cc:1397] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260928 16:43:37.501257 12083 raft_consensus.cc:348] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141d428d501f4a12a7458e788e1ed9f1" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33623 } } peers { permanent_uuid: "496139b1aa904b1eb6a9f6cefaa16ced" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 36623 } } peers { permanent_uuid: "8338b65caaf949c2bac7329fce4db370" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37423 } }
I20260928 16:43:37.501619 12083 raft_consensus.cc:374] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:37.501686 12083 raft_consensus.cc:729] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 496139b1aa904b1eb6a9f6cefaa16ced, State: Initialized, Role: FOLLOWER
I20260928 16:43:37.501618 12082 raft_consensus.cc:348] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141d428d501f4a12a7458e788e1ed9f1" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33623 } } peers { permanent_uuid: "496139b1aa904b1eb6a9f6cefaa16ced" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 36623 } } peers { permanent_uuid: "8338b65caaf949c2bac7329fce4db370" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37423 } }
I20260928 16:43:37.502003 12082 raft_consensus.cc:374] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:37.502074 12082 raft_consensus.cc:729] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 141d428d501f4a12a7458e788e1ed9f1, State: Initialized, Role: FOLLOWER
I20260928 16:43:37.501978 12084 raft_consensus.cc:348] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141d428d501f4a12a7458e788e1ed9f1" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33623 } } peers { permanent_uuid: "496139b1aa904b1eb6a9f6cefaa16ced" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 36623 } } peers { permanent_uuid: "8338b65caaf949c2bac7329fce4db370" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37423 } }
I20260928 16:43:37.502283 12084 raft_consensus.cc:374] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:37.502336 12084 raft_consensus.cc:729] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8338b65caaf949c2bac7329fce4db370, State: Initialized, Role: FOLLOWER
I20260928 16:43:37.502257 12083 consensus_queue.cc:260] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141d428d501f4a12a7458e788e1ed9f1" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33623 } } peers { permanent_uuid: "496139b1aa904b1eb6a9f6cefaa16ced" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 36623 } } peers { permanent_uuid: "8338b65caaf949c2bac7329fce4db370" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37423 } }
I20260928 16:43:37.502713 12082 consensus_queue.cc:260] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141d428d501f4a12a7458e788e1ed9f1" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33623 } } peers { permanent_uuid: "496139b1aa904b1eb6a9f6cefaa16ced" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 36623 } } peers { permanent_uuid: "8338b65caaf949c2bac7329fce4db370" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37423 } }
I20260928 16:43:37.502815 12084 consensus_queue.cc:260] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141d428d501f4a12a7458e788e1ed9f1" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33623 } } peers { permanent_uuid: "496139b1aa904b1eb6a9f6cefaa16ced" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 36623 } } peers { permanent_uuid: "8338b65caaf949c2bac7329fce4db370" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37423 } }
I20260928 16:43:37.503473 12083 ts_tablet_manager.cc:1428] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 16:43:37.503633 11796 heartbeater.cc:503] Master 127.25.71.126:37147 was elected leader, sending a full tablet report...
I20260928 16:43:37.503722 12084 ts_tablet_manager.cc:1428] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:43:37.503885 12065 heartbeater.cc:503] Master 127.25.71.126:37147 was elected leader, sending a full tablet report...
I20260928 16:43:37.504287 11931 heartbeater.cc:503] Master 127.25.71.126:37147 was elected leader, sending a full tablet report...
I20260928 16:43:37.505506 12082 ts_tablet_manager.cc:1428] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1: Time spent starting tablet: real 0.009s user 0.002s sys 0.005s
W20260928 16:43:37.635653 11797 tablet.cc:2369] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:43:37.648054 11932 tablet.cc:2369] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:43:37.648054 12066 tablet.cc:2369] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:43:37.698668 12089 raft_consensus.cc:482] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:37.698968 12089 raft_consensus.cc:504] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141d428d501f4a12a7458e788e1ed9f1" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33623 } } peers { permanent_uuid: "496139b1aa904b1eb6a9f6cefaa16ced" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 36623 } } peers { permanent_uuid: "8338b65caaf949c2bac7329fce4db370" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37423 } }
I20260928 16:43:37.700198 12089 leader_election.cc:288] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 496139b1aa904b1eb6a9f6cefaa16ced (127.25.71.67:36623), 8338b65caaf949c2bac7329fce4db370 (127.25.71.66:37423)
I20260928 16:43:37.706063 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:37.701624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:47725 (local address 127.25.71.66:37423)
0928 16:43:37.701774 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:37.701777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:37.701877 (+ 100us) server_negotiation.cc:407] Connection header received
0928 16:43:37.702049 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:37.702052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:37.702106 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:37.702213 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:37.702737 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.703321 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.704927 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.705362 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.705482 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:37.705543 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:37.705820 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 16:43:37.705907 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:43:37.706112 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:37.700826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:60845 (local address 127.25.71.67:36623)
0928 16:43:37.700968 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:37.700972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:37.701319 (+ 347us) server_negotiation.cc:407] Connection header received
0928 16:43:37.701714 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:37.701719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:37.701803 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:37.701894 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:37.702345 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.703143 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.704918 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.705411 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.705549 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:37.705631 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:37.705844 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 16:43:37.705934 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:43:37.706171 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:37.701323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:37423 (local address 127.25.71.65:47725)
0928 16:43:37.701753 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:37.701770 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:37.701931 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:37.702369 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:37.702373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:37.702400 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:37.702546 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:37.702552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.703499 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.703503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:37.704751 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:37.704764 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.705503 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.705507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:37.705667 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:37.705692 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 16:43:37.705772 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 16:43:37.705943 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":130,"thread_start_us":122,"threads_started":1}
I20260928 16:43:37.706449 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:37.700812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:36623 (local address 127.25.71.65:60845)
0928 16:43:37.701184 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:37.701200 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:37.701362 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:37.701943 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:37.701955 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:37.701986 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:37.702168 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:37.702178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.703295 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.703300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:37.704730 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:37.704744 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.705578 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.705582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:37.705701 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:37.705728 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:43:37.705815 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 16:43:37.706020 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":189,"thread_start_us":141,"threads_started":1}
I20260928 16:43:37.707242 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5254d4c43bb346809a3a212e691f7d9c" candidate_uuid: "141d428d501f4a12a7458e788e1ed9f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8338b65caaf949c2bac7329fce4db370" is_pre_election: true
I20260928 16:43:37.707594 11885 raft_consensus.cc:2454] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 141d428d501f4a12a7458e788e1ed9f1 in term 0.
I20260928 16:43:37.707579 12019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5254d4c43bb346809a3a212e691f7d9c" candidate_uuid: "141d428d501f4a12a7458e788e1ed9f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "496139b1aa904b1eb6a9f6cefaa16ced" is_pre_election: true
I20260928 16:43:37.707937 12019 raft_consensus.cc:2454] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 141d428d501f4a12a7458e788e1ed9f1 in term 0.
I20260928 16:43:37.708308 11684 leader_election.cc:302] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 141d428d501f4a12a7458e788e1ed9f1, 8338b65caaf949c2bac7329fce4db370; no voters:
I20260928 16:43:37.708702 12089 raft_consensus.cc:2781] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:43:37.708874 12089 raft_consensus.cc:482] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:43:37.709012 12089 raft_consensus.cc:3037] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:37.710883 12089 raft_consensus.cc:504] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141d428d501f4a12a7458e788e1ed9f1" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33623 } } peers { permanent_uuid: "496139b1aa904b1eb6a9f6cefaa16ced" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 36623 } } peers { permanent_uuid: "8338b65caaf949c2bac7329fce4db370" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37423 } }
I20260928 16:43:37.711336 12089 leader_election.cc:288] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1 [CANDIDATE]: Term 1 election: Requested vote from peers 496139b1aa904b1eb6a9f6cefaa16ced (127.25.71.67:36623), 8338b65caaf949c2bac7329fce4db370 (127.25.71.66:37423)
I20260928 16:43:37.712110 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5254d4c43bb346809a3a212e691f7d9c" candidate_uuid: "141d428d501f4a12a7458e788e1ed9f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8338b65caaf949c2bac7329fce4db370"
I20260928 16:43:37.712157 12019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5254d4c43bb346809a3a212e691f7d9c" candidate_uuid: "141d428d501f4a12a7458e788e1ed9f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "496139b1aa904b1eb6a9f6cefaa16ced"
I20260928 16:43:37.712280 11885 raft_consensus.cc:3037] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:37.712329 12019 raft_consensus.cc:3037] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:37.713686 12019 raft_consensus.cc:2454] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 141d428d501f4a12a7458e788e1ed9f1 in term 1.
I20260928 16:43:37.713686 11885 raft_consensus.cc:2454] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 141d428d501f4a12a7458e788e1ed9f1 in term 1.
I20260928 16:43:37.714303 11684 leader_election.cc:302] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 141d428d501f4a12a7458e788e1ed9f1, 496139b1aa904b1eb6a9f6cefaa16ced; no voters:
I20260928 16:43:37.714680 12089 raft_consensus.cc:2781] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:37.714946 12089 raft_consensus.cc:686] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1 [term 1 LEADER]: Becoming Leader. State: Replica: 141d428d501f4a12a7458e788e1ed9f1, State: Running, Role: LEADER
I20260928 16:43:37.715374 12089 consensus_queue.cc:237] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141d428d501f4a12a7458e788e1ed9f1" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33623 } } peers { permanent_uuid: "496139b1aa904b1eb6a9f6cefaa16ced" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 36623 } } peers { permanent_uuid: "8338b65caaf949c2bac7329fce4db370" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37423 } }
I20260928 16:43:37.720268 11604 catalog_manager.cc:5495] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 141d428d501f4a12a7458e788e1ed9f1 (127.25.71.65). New cstate: current_term: 1 leader_uuid: "141d428d501f4a12a7458e788e1ed9f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141d428d501f4a12a7458e788e1ed9f1" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 33623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "496139b1aa904b1eb6a9f6cefaa16ced" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 36623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8338b65caaf949c2bac7329fce4db370" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37423 } health_report { overall_health: UNKNOWN } } }
I20260928 16:43:37.789294 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:37.784099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49414 (local address 127.25.71.65:33623)
0928 16:43:37.784282 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:37.784288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:37.784440 (+ 152us) server_negotiation.cc:407] Connection header received
0928 16:43:37.784725 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:37.784728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:37.784808 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:37.784940 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:37.785311 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.785956 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.787219 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.787528 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.787670 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:37.788110 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:37.788697 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:37.788832 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:37.789007 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 16:43:37.789075 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:43:37.799362 12019 raft_consensus.cc:1261] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced [term 1 FOLLOWER]: Refusing update from remote peer 141d428d501f4a12a7458e788e1ed9f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:37.799362 11885 raft_consensus.cc:1261] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370 [term 1 FOLLOWER]: Refusing update from remote peer 141d428d501f4a12a7458e788e1ed9f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:37.800607 12089 consensus_queue.cc:1035] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8338b65caaf949c2bac7329fce4db370" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 37423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:37.801033 12093 consensus_queue.cc:1035] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "496139b1aa904b1eb6a9f6cefaa16ced" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 36623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:37.813155 12098 mvcc.cc:205] Tried to move back new op lower bound from 7334354197693190144 to 7334354197365710848. Current Snapshot: MvccSnapshot[applied={T|T < 7334354197693190144}]
I20260928 16:43:37.813802 12097 mvcc.cc:205] Tried to move back new op lower bound from 7334354197693190144 to 7334354197365710848. Current Snapshot: MvccSnapshot[applied={T|T < 7334354197693190144}]
I20260928 16:43:37.823693 11604 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:37.823917 11604 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:37.827363 11604 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2129bb0743fd46d4b3a973864dd2dc9c: Sending DeleteTablet for 3 replicas of tablet 5254d4c43bb346809a3a212e691f7d9c
I20260928 16:43:37.828442 11999 tablet_service.cc:1514] Processing DeleteTablet for tablet 5254d4c43bb346809a3a212e691f7d9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:37 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:57386
I20260928 16:43:37.828506 11865 tablet_service.cc:1514] Processing DeleteTablet for tablet 5254d4c43bb346809a3a212e691f7d9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:37 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:35178
I20260928 16:43:37.828702 11730 tablet_service.cc:1514] Processing DeleteTablet for tablet 5254d4c43bb346809a3a212e691f7d9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:37 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:49400
I20260928 16:43:37.829195 12106 tablet_replica.cc:330] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370: stopping tablet replica
I20260928 16:43:37.829349 12105 tablet_replica.cc:330] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced: stopping tablet replica
I20260928 16:43:37.829728 12107 tablet_replica.cc:330] T 5254d4c43bb346809a3a212e691f7d9c P 141d428d501f4a12a7458e788e1ed9f1: stopping tablet replica
I20260928 16:43:37.829862 12105 raft_consensus.cc:2229] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:37.830240 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 11667
I20260928 16:43:37.830271 12105 raft_consensus.cc:2258] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:37.831153 12106 raft_consensus.cc:2229] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:37.832293 12105 ts_tablet_manager.cc:1905] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:37.832669 12106 raft_consensus.cc:2258] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:37.835125 12105 ts_tablet_manager.cc:1918] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:43:37.835243 12105 log.cc:1196] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/5254d4c43bb346809a3a212e691f7d9c
I20260928 16:43:37.835184 12106 ts_tablet_manager.cc:1905] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:37.835580 12105 ts_tablet_manager.cc:1939] T 5254d4c43bb346809a3a212e691f7d9c P 496139b1aa904b1eb6a9f6cefaa16ced: Deleting consensus metadata
I20260928 16:43:37.836897 11590 catalog_manager.cc:4841] TS 496139b1aa904b1eb6a9f6cefaa16ced (127.25.71.67:36623): tablet 5254d4c43bb346809a3a212e691f7d9c (table test-table [id=6d270f2fe3e34f47b341d52b8dc13462]) successfully deleted
I20260928 16:43:37.839411 12106 ts_tablet_manager.cc:1918] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:43:37.839591 12106 log.cc:1196] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/5254d4c43bb346809a3a212e691f7d9c
I20260928 16:43:37.840020 12106 ts_tablet_manager.cc:1939] T 5254d4c43bb346809a3a212e691f7d9c P 8338b65caaf949c2bac7329fce4db370: Deleting consensus metadata
W20260928 16:43:37.841259 11589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.71.65:33623 (error 108)
I20260928 16:43:37.841774 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 11801
I20260928 16:43:37.843216 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:37.842736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:33623 (local address 127.0.0.1:49426)
0928 16:43:37.843016 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:37.843097 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.65:33623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
W20260928 16:43:37.843407 11589 catalog_manager.cc:4584] TS 141d428d501f4a12a7458e788e1ed9f1 (127.25.71.65:33623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5254d4c43bb346809a3a212e691f7d9c: Network error: Client connection negotiation failed: client connection to 127.25.71.65:33623: connect: Connection refused (error 111)
I20260928 16:43:37.849128 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 11936
I20260928 16:43:37.851480 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:37.850958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:37423 (local address 127.0.0.1:35180)
0928 16:43:37.851270 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:37.851356 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.66:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260928 16:43:37.855531 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 11573
2026-09-28T16:43:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2645 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T16:43:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:37Z Disabled control of system clock
I20260928 16:43:37.892434 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:42531
--webserver_interface=127.25.71.126
--webserver_port=0
--builtin_ntp_servers=127.25.71.84:43099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:42531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:43:38.028609 12112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:38.029022 12112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:38.029070 12112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:38.034116 12112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:38.034256 12112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:38.034273 12112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:38.034292 12112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:38.034308 12112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:38.040155 12112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:42531
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:42531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:38.041810 12112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:38.043727 12112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:38.054527 12118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:38.054510 12117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:38.054548 12120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:38.054913 12112 server_base.cc:1034] running on GCE node
I20260928 16:43:38.055943 12112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:38.057799 12112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:38.059042 12112 hybrid_clock.cc:648] HybridClock initialized: now 1790613818059006 us; error 62 us; skew 500 ppm
I20260928 16:43:38.061904 12112 webserver.cc:460] Webserver started at http://127.25.71.126:36801/ using document root <none> and password file <none>
I20260928 16:43:38.062762 12112 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:38.062867 12112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:38.063143 12112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:38.065882 12112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "52470194090741feb8547a865ef301e9"
format_stamp: "Formatted at 2026-09-28 16:43:38 on dist-test-slave-nqgh"
server_key: "8857c75a7397c6582ed8bd3a941563e1"
server_key_iv: "203ab723cff51a7df94de01d07601bfc"
server_key_version: "encryptionkey@0"
I20260928 16:43:38.066571 12112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "52470194090741feb8547a865ef301e9"
format_stamp: "Formatted at 2026-09-28 16:43:38 on dist-test-slave-nqgh"
server_key: "8857c75a7397c6582ed8bd3a941563e1"
server_key_iv: "203ab723cff51a7df94de01d07601bfc"
server_key_version: "encryptionkey@0"
I20260928 16:43:38.071837 12112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 16:43:38.075598 12126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:38.077690 12112 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:43:38.077946 12112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "52470194090741feb8547a865ef301e9"
format_stamp: "Formatted at 2026-09-28 16:43:38 on dist-test-slave-nqgh"
server_key: "8857c75a7397c6582ed8bd3a941563e1"
server_key_iv: "203ab723cff51a7df94de01d07601bfc"
server_key_version: "encryptionkey@0"
I20260928 16:43:38.078115 12112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:38.097572 12112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:38.098891 12112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:38.099190 12112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:38.110098 12179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:42531 every 8 connection(s)
I20260928 16:43:38.110069 12112 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:42531
I20260928 16:43:38.111896 12112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:38.116092 12180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:38.122080 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 12112
I20260928 16:43:38.122313 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:43:38.122889 25885 external_mini_cluster.cc:1428] Setting key a27ded7059bdec7204f29710be3f49cb
I20260928 16:43:38.122959 12180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9: Bootstrap starting.
I20260928 16:43:38.125984 12180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:38.126921 12180 log.cc:824] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:38.129913 12180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9: No bootstrap required, opened a new log
I20260928 16:43:38.135802 12180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52470194090741feb8547a865ef301e9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42531 } }
I20260928 16:43:38.136168 12180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:38.136224 12180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52470194090741feb8547a865ef301e9, State: Initialized, Role: FOLLOWER
I20260928 16:43:38.136807 12180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52470194090741feb8547a865ef301e9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42531 } }
I20260928 16:43:38.136978 12180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:38.137053 12180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:38.137164 12180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:38.138139 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:38.124912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43386 (local address 127.25.71.126:42531)
0928 16:43:38.125897 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:38.125918 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:38.125974 (+ 56us) server_negotiation.cc:407] Connection header received
0928 16:43:38.127306 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:38.127342 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:38.127932 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:38.128504 (+ 572us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:38.129121 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.130428 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.131996 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.132849 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.133080 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:38.134944 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:38.134956 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:38.134994 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:38.135841 (+ 847us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:38.135953 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:38.135991 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:38.136132 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:38.136416 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 16:43:38.136900 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":188,"threads_started":1}
I20260928 16:43:38.138294 12180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52470194090741feb8547a865ef301e9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42531 } }
I20260928 16:43:38.138780 12180 leader_election.cc:302] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52470194090741feb8547a865ef301e9; no voters:
I20260928 16:43:38.139180 12180 leader_election.cc:288] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:38.139295 12185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:38.139479 12185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9 [term 1 LEADER]: Becoming Leader. State: Replica: 52470194090741feb8547a865ef301e9, State: Running, Role: LEADER
I20260928 16:43:38.139835 12185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52470194090741feb8547a865ef301e9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42531 } }
I20260928 16:43:38.140739 12180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:38.142125 12185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52470194090741feb8547a865ef301e9. Latest consensus state: current_term: 1 leader_uuid: "52470194090741feb8547a865ef301e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52470194090741feb8547a865ef301e9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42531 } } }
I20260928 16:43:38.142374 12185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:38.142426 12186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52470194090741feb8547a865ef301e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52470194090741feb8547a865ef301e9" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 42531 } } }
I20260928 16:43:38.142554 12186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:38.143162 12189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:38.147249 12189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:38.155814 12189 catalog_manager.cc:1347] Generated new cluster ID: 11db92f1a95f4133aaada65f94513548
I20260928 16:43:38.155942 12189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:38.169041 12189 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:38.170140 12189 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:38.185215 12189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9: Generated new TSK 0
I20260928 16:43:38.186041 12189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:43:38.193821 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:42531
--builtin_ntp_servers=127.25.71.84:43099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:43:38.321341 12204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:38.321681 12204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:38.321725 12204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:38.326079 12204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:38.326167 12204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:38.326228 12204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:43:38.331508 12204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:42531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:38.332829 12204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:38.334316 12204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:43:38.346086 12204 server_base.cc:1034] running on GCE node
W20260928 16:43:38.346072 12212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:38.346148 12210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:38.346112 12209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:38.346982 12204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:38.347760 12204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:38.348990 12204 hybrid_clock.cc:648] HybridClock initialized: now 1790613818348963 us; error 66 us; skew 500 ppm
I20260928 16:43:38.351542 12204 webserver.cc:460] Webserver started at http://127.25.71.65:38113/ using document root <none> and password file <none>
I20260928 16:43:38.352303 12204 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:38.352393 12204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:38.352612 12204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:38.354914 12204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "6c77f347cb214958920c6d75fc7735c8"
format_stamp: "Formatted at 2026-09-28 16:43:38 on dist-test-slave-nqgh"
server_key: "4e47623b3f6883a85c0134265f9e466b"
server_key_iv: "79c26146da79a263f65ac95edfb74c47"
server_key_version: "encryptionkey@0"
I20260928 16:43:38.355535 12204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "6c77f347cb214958920c6d75fc7735c8"
format_stamp: "Formatted at 2026-09-28 16:43:38 on dist-test-slave-nqgh"
server_key: "4e47623b3f6883a85c0134265f9e466b"
server_key_iv: "79c26146da79a263f65ac95edfb74c47"
server_key_version: "encryptionkey@0"
I20260928 16:43:38.360293 12204 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:43:38.364017 12218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:38.366122 12204 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:43:38.366324 12204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "6c77f347cb214958920c6d75fc7735c8"
format_stamp: "Formatted at 2026-09-28 16:43:38 on dist-test-slave-nqgh"
server_key: "4e47623b3f6883a85c0134265f9e466b"
server_key_iv: "79c26146da79a263f65ac95edfb74c47"
server_key_version: "encryptionkey@0"
I20260928 16:43:38.366475 12204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:38.383100 12204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:38.383863 12204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:38.384087 12204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:38.384724 12204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:38.385751 12204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:38.385802 12204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:38.385859 12204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:38.385888 12204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:38.396472 12204 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:40183
I20260928 16:43:38.396555 12333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:40183 every 8 connection(s)
I20260928 16:43:38.397557 12204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:43:38.403388 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 12204
I20260928 16:43:38.403574 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:43:38.403997 25885 external_mini_cluster.cc:1428] Setting key 646d48111542a982762b1e0c75b46c41
I20260928 16:43:38.406491 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:42531
--builtin_ntp_servers=127.25.71.84:43099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:43:38.411559 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:38.400137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:34495 (local address 127.25.71.126:42531)
0928 16:43:38.400412 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:38.400416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:38.401845 (+ 1429us) server_negotiation.cc:407] Connection header received
0928 16:43:38.404106 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:38.404110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:38.404197 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:38.404308 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:38.405943 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.406750 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.408286 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.408647 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.408781 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:38.410692 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:38.410697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:38.410701 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:38.410725 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:38.410761 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:38.410775 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:38.410875 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:38.411281 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 16:43:38.411374 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260928 16:43:38.413107 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:38.400530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:42531 (local address 127.25.71.65:34495)
0928 16:43:38.401567 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:38.401669 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:38.403676 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:38.404609 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:38.404619 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:38.404790 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:38.405715 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:38.405738 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.406946 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.406951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:38.408107 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:38.408123 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.408911 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.408916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:38.409212 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:38.410105 (+ 893us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:43:38.410107 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:38.410190 (+ 83us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:43:38.410331 (+ 141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:38.411015 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:38.411051 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:38.411063 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 16:43:38.411822 (+ 759us) client_negotiation.cc:240] Negotiation successful
0928 16:43:38.412209 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":142,"threads_started":1}
I20260928 16:43:38.414920 12334 heartbeater.cc:347] Connected to a master server at 127.25.71.126:42531
I20260928 16:43:38.415386 12334 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:38.416412 12334 heartbeater.cc:511] Master 127.25.71.126:42531 requested a full tablet report, sending...
I20260928 16:43:38.419209 12143 ts_manager.cc:194] Registered new tserver with Master: 6c77f347cb214958920c6d75fc7735c8 (127.25.71.65:40183)
I20260928 16:43:38.421207 12143 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.71.65:34495
W20260928 16:43:38.551427 12337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:38.551841 12337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:38.551887 12337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:38.556593 12337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:38.556702 12337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:38.556771 12337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:43:38.562561 12337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:42531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:38.564203 12337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:38.566025 12337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:38.578856 12343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:38.578845 12345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:38.578945 12337 server_base.cc:1034] running on GCE node
W20260928 16:43:38.578845 12342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:38.579773 12337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:38.580652 12337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:38.581892 12337 hybrid_clock.cc:648] HybridClock initialized: now 1790613818581848 us; error 58 us; skew 500 ppm
I20260928 16:43:38.584651 12337 webserver.cc:460] Webserver started at http://127.25.71.66:41139/ using document root <none> and password file <none>
I20260928 16:43:38.585352 12337 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:38.585446 12337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:38.585682 12337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:38.588022 12337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "8fa70358703847a8b18809135dd09d18"
format_stamp: "Formatted at 2026-09-28 16:43:38 on dist-test-slave-nqgh"
server_key: "071753d768fced1a3dc2f401445d4b1c"
server_key_iv: "a5767696991c3860de503e2768d2bf94"
server_key_version: "encryptionkey@0"
I20260928 16:43:38.588640 12337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "8fa70358703847a8b18809135dd09d18"
format_stamp: "Formatted at 2026-09-28 16:43:38 on dist-test-slave-nqgh"
server_key: "071753d768fced1a3dc2f401445d4b1c"
server_key_iv: "a5767696991c3860de503e2768d2bf94"
server_key_version: "encryptionkey@0"
I20260928 16:43:38.593178 12337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:43:38.596392 12351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:38.598137 12337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:43:38.598297 12337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "8fa70358703847a8b18809135dd09d18"
format_stamp: "Formatted at 2026-09-28 16:43:38 on dist-test-slave-nqgh"
server_key: "071753d768fced1a3dc2f401445d4b1c"
server_key_iv: "a5767696991c3860de503e2768d2bf94"
server_key_version: "encryptionkey@0"
I20260928 16:43:38.598434 12337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:38.612843 12337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:38.613626 12337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:38.613812 12337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:38.614436 12337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:38.615541 12337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:38.615594 12337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:38.615651 12337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:38.615679 12337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:38.626108 12337 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:36127
I20260928 16:43:38.626212 12465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:36127 every 8 connection(s)
I20260928 16:43:38.627202 12337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:43:38.636428 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 12337
I20260928 16:43:38.636602 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:43:38.637024 25885 external_mini_cluster.cc:1428] Setting key 2d3d79fd42d6c73017e8de2b6e776136
I20260928 16:43:38.640055 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:42531
--builtin_ntp_servers=127.25.71.84:43099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:43:38.640650 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:38.629727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:49437 (local address 127.25.71.126:42531)
0928 16:43:38.629957 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:38.629963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:38.631521 (+ 1558us) server_negotiation.cc:407] Connection header received
0928 16:43:38.633303 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:38.633307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:38.633385 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:38.633502 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:38.635145 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.635729 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.637851 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.638118 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.638236 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:38.640055 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:38.640058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:38.640060 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:38.640076 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:38.640099 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:38.640108 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:38.640174 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:38.640399 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 16:43:38.640484 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 16:43:38.641655 12468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:38.630170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:42531 (local address 127.25.71.66:49437)
0928 16:43:38.631278 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:38.631346 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:38.632946 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:38.633830 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:38.633841 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:38.633974 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:38.634893 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:38.634918 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.635935 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.635939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:38.637647 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:38.637665 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.638367 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.638373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:38.638607 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:38.639616 (+ 1009us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:43:38.639618 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:38.639674 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:43:38.639774 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:38.640195 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:38.640206 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:38.640211 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:43:38.640424 (+ 213us) client_negotiation.cc:240] Negotiation successful
0928 16:43:38.640794 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":142,"threads_started":1}
I20260928 16:43:38.643235 12466 heartbeater.cc:347] Connected to a master server at 127.25.71.126:42531
I20260928 16:43:38.643512 12466 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:38.644133 12466 heartbeater.cc:511] Master 127.25.71.126:42531 requested a full tablet report, sending...
I20260928 16:43:38.645807 12143 ts_manager.cc:194] Registered new tserver with Master: 8fa70358703847a8b18809135dd09d18 (127.25.71.66:36127)
I20260928 16:43:38.646852 12143 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.71.66:49437
W20260928 16:43:38.787318 12469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:38.787716 12469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:38.787765 12469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:38.792271 12469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:38.792389 12469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:38.792459 12469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:43:38.797992 12469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:43099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:42531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:38.799607 12469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:38.801409 12469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:38.815021 12477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:38.815028 12474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:38.815125 12469 server_base.cc:1034] running on GCE node
W20260928 16:43:38.815052 12475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:38.815910 12469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:38.816651 12469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:38.817840 12469 hybrid_clock.cc:648] HybridClock initialized: now 1790613818817825 us; error 43 us; skew 500 ppm
I20260928 16:43:38.820267 12469 webserver.cc:460] Webserver started at http://127.25.71.67:43579/ using document root <none> and password file <none>
I20260928 16:43:38.820955 12469 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:38.821031 12469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:38.821264 12469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:38.823693 12469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "b3ca973be5a54d8ba5c337f1a0c1b2f4"
format_stamp: "Formatted at 2026-09-28 16:43:38 on dist-test-slave-nqgh"
server_key: "1ddbf2902e553cec66de0f40b744165c"
server_key_iv: "617dc9566f48dc928cde2bb43a89de68"
server_key_version: "encryptionkey@0"
I20260928 16:43:38.824316 12469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "b3ca973be5a54d8ba5c337f1a0c1b2f4"
format_stamp: "Formatted at 2026-09-28 16:43:38 on dist-test-slave-nqgh"
server_key: "1ddbf2902e553cec66de0f40b744165c"
server_key_iv: "617dc9566f48dc928cde2bb43a89de68"
server_key_version: "encryptionkey@0"
I20260928 16:43:38.829099 12469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 16:43:38.832505 12483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:38.834347 12469 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 16:43:38.834539 12469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "b3ca973be5a54d8ba5c337f1a0c1b2f4"
format_stamp: "Formatted at 2026-09-28 16:43:38 on dist-test-slave-nqgh"
server_key: "1ddbf2902e553cec66de0f40b744165c"
server_key_iv: "617dc9566f48dc928cde2bb43a89de68"
server_key_version: "encryptionkey@0"
I20260928 16:43:38.834743 12469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:38.863255 12469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:38.864018 12469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:38.864199 12469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:38.864850 12469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:38.865919 12469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:38.865972 12469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:38.866030 12469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:38.866057 12469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:38.877516 12469 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:43835
I20260928 16:43:38.877586 12597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:43835 every 8 connection(s)
I20260928 16:43:38.878765 12469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:43:38.879932 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 12469
I20260928 16:43:38.880054 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:43:38.880364 25885 external_mini_cluster.cc:1428] Setting key 37f1d8ba047f16c64cf4256a9d6e3c76
I20260928 16:43:38.889858 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:38.881292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:32999 (local address 127.25.71.126:42531)
0928 16:43:38.881471 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:38.881477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:38.883136 (+ 1659us) server_negotiation.cc:407] Connection header received
0928 16:43:38.884740 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:38.884744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:38.884810 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:38.884887 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:38.885860 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.886425 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.887681 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.887916 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.888008 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:38.889310 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:38.889313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:38.889315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:38.889368 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:38.889388 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:38.889396 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:38.889458 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:38.889621 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 16:43:38.889697 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:43:38.890828 12600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:38.881969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:42531 (local address 127.25.71.67:32999)
0928 16:43:38.882930 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:38.882995 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:38.884498 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:38.885089 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:38.885097 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:38.885192 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:38.885714 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:38.885727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.886552 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.886555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:38.887553 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:38.887565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.888083 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.888088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:38.888250 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:38.889018 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:43:38.889020 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:38.889065 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:43:38.889140 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:38.889477 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:38.889488 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:38.889493 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:43:38.889613 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 16:43:38.889942 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":151,"threads_started":1}
I20260928 16:43:38.892354 12598 heartbeater.cc:347] Connected to a master server at 127.25.71.126:42531
I20260928 16:43:38.892763 12598 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:38.893620 12598 heartbeater.cc:511] Master 127.25.71.126:42531 requested a full tablet report, sending...
I20260928 16:43:38.895358 12143 ts_manager.cc:194] Registered new tserver with Master: b3ca973be5a54d8ba5c337f1a0c1b2f4 (127.25.71.67:43835)
I20260928 16:43:38.896148 12143 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.71.67:32999
I20260928 16:43:38.896210 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:43:38.903708 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:38.898832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43392 (local address 127.25.71.126:42531)
0928 16:43:38.899029 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:38.899033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:38.899334 (+ 301us) server_negotiation.cc:407] Connection header received
0928 16:43:38.899406 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:38.899409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:38.899471 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:38.899578 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:38.900054 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.900644 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.901761 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.902009 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.902104 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:38.903171 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:38.903174 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:38.903177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:38.903194 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:38.903221 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:38.903229 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:38.903353 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:38.903457 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 16:43:38.903507 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:43:38.907554 12143 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:43:38.911131 12143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:43:38.929679 12610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:38.922788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:36127 (local address 127.0.0.1:44248)
0928 16:43:38.923675 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:38.923758 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:38.923951 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:38.925102 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:38.925105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:38.925125 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:38.925240 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:38.925247 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.926384 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.926388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:38.927920 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:38.927932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.928953 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.928957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:38.929068 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:38.929109 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 16:43:38.929240 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 16:43:38.929385 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":3,"spinlock_wait_cycles":3968,"thread_start_us":149,"threads_started":1}
I20260928 16:43:38.929852 12613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:38.923509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.25.71.65:40183)
0928 16:43:38.924160 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:38.924168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:38.924436 (+ 268us) server_negotiation.cc:407] Connection header received
0928 16:43:38.924681 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:38.924688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:38.924936 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:38.925069 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:38.925535 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.926718 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.928149 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.928984 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.929111 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:38.929366 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:38.929515 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 16:43:38.929670 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":67,"threads_started":1}
I20260928 16:43:38.930195 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:38.922930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:40183 (local address 127.0.0.1:42676)
0928 16:43:38.924326 (+ 1396us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:38.924351 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:38.924519 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:38.925238 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:38.925241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:38.925259 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:38.925373 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:38.925379 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.926851 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.926855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:38.928000 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:38.928011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.929253 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.929257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:38.929355 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:38.929377 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 16:43:38.929468 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 16:43:38.929694 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1326,"mutex_wait_us":1,"thread_start_us":144,"threads_started":1}
I20260928 16:43:38.930231 12614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:38.923556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.25.71.66:36127)
0928 16:43:38.924290 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:38.924304 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:38.924334 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:43:38.924425 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:38.924431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:38.924702 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:38.924889 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:38.925410 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.926242 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.928091 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.928822 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.928931 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:38.929161 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:38.929725 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 16:43:38.930075 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":149,"threads_started":1}
I20260928 16:43:38.930544 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:38.922512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:43835 (local address 127.0.0.1:36056)
0928 16:43:38.923675 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:38.923758 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:38.923951 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:38.924915 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:38.924922 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:38.924946 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:38.925119 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:38.925128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.926963 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.926967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:38.928288 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:38.928297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.929544 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.929547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:38.929793 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:38.929820 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:43:38.930334 (+ 514us) client_negotiation.cc:240] Negotiation successful
0928 16:43:38.930422 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":134,"spinlock_wait_cycles":8448,"thread_start_us":137,"threads_started":1}
I20260928 16:43:38.930866 12611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:38.922796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36056 (local address 127.25.71.67:43835)
0928 16:43:38.924289 (+ 1493us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:38.924304 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:38.924334 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:43:38.924425 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:38.924431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:38.924702 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:38.924889 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:38.925262 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.926589 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.928428 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:38.929187 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:38.929290 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:38.929455 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:38.930516 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0928 16:43:38.930679 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1371,"thread_start_us":182,"threads_started":1}
I20260928 16:43:38.933902 12399 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8d104faee34330a6ce226d9bcccd78 (DEFAULT_TABLE table=test-table [id=956202f39a534c4d8ce79e9afd0c8303]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:38.933903 12267 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8d104faee34330a6ce226d9bcccd78 (DEFAULT_TABLE table=test-table [id=956202f39a534c4d8ce79e9afd0c8303]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:38.934002 12531 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8d104faee34330a6ce226d9bcccd78 (DEFAULT_TABLE table=test-table [id=956202f39a534c4d8ce79e9afd0c8303]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:38.935316 12531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8d104faee34330a6ce226d9bcccd78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:38.935310 12267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8d104faee34330a6ce226d9bcccd78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:38.935310 12399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8d104faee34330a6ce226d9bcccd78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:38.945683 12617 tablet_bootstrap.cc:492] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4: Bootstrap starting.
I20260928 16:43:38.945683 12615 tablet_bootstrap.cc:492] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18: Bootstrap starting.
I20260928 16:43:38.945850 12616 tablet_bootstrap.cc:492] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8: Bootstrap starting.
I20260928 16:43:38.948400 12616 tablet_bootstrap.cc:654] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:38.948416 12615 tablet_bootstrap.cc:654] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:38.948416 12617 tablet_bootstrap.cc:654] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:38.949337 12616 log.cc:824] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:38.949388 12617 log.cc:824] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:38.949397 12615 log.cc:824] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:38.952184 12616 tablet_bootstrap.cc:492] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8: No bootstrap required, opened a new log
I20260928 16:43:38.952180 12617 tablet_bootstrap.cc:492] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4: No bootstrap required, opened a new log
I20260928 16:43:38.952440 12617 ts_tablet_manager.cc:1397] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260928 16:43:38.952440 12616 ts_tablet_manager.cc:1397] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 16:43:38.952584 12615 tablet_bootstrap.cc:492] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18: No bootstrap required, opened a new log
I20260928 16:43:38.952813 12615 ts_tablet_manager.cc:1397] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:43:38.956665 12615 raft_consensus.cc:348] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ca973be5a54d8ba5c337f1a0c1b2f4" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43835 } } peers { permanent_uuid: "8fa70358703847a8b18809135dd09d18" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36127 } } peers { permanent_uuid: "6c77f347cb214958920c6d75fc7735c8" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 40183 } }
I20260928 16:43:38.957012 12615 raft_consensus.cc:374] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:38.957082 12615 raft_consensus.cc:729] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fa70358703847a8b18809135dd09d18, State: Initialized, Role: FOLLOWER
I20260928 16:43:38.957273 12617 raft_consensus.cc:348] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ca973be5a54d8ba5c337f1a0c1b2f4" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43835 } } peers { permanent_uuid: "8fa70358703847a8b18809135dd09d18" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36127 } } peers { permanent_uuid: "6c77f347cb214958920c6d75fc7735c8" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 40183 } }
I20260928 16:43:38.957556 12616 raft_consensus.cc:348] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ca973be5a54d8ba5c337f1a0c1b2f4" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43835 } } peers { permanent_uuid: "8fa70358703847a8b18809135dd09d18" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36127 } } peers { permanent_uuid: "6c77f347cb214958920c6d75fc7735c8" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 40183 } }
I20260928 16:43:38.957660 12615 consensus_queue.cc:260] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ca973be5a54d8ba5c337f1a0c1b2f4" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43835 } } peers { permanent_uuid: "8fa70358703847a8b18809135dd09d18" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36127 } } peers { permanent_uuid: "6c77f347cb214958920c6d75fc7735c8" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 40183 } }
I20260928 16:43:38.957780 12616 raft_consensus.cc:374] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:38.957782 12617 raft_consensus.cc:374] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:38.957862 12617 raft_consensus.cc:729] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ca973be5a54d8ba5c337f1a0c1b2f4, State: Initialized, Role: FOLLOWER
I20260928 16:43:38.957850 12616 raft_consensus.cc:729] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c77f347cb214958920c6d75fc7735c8, State: Initialized, Role: FOLLOWER
I20260928 16:43:38.958483 12616 consensus_queue.cc:260] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ca973be5a54d8ba5c337f1a0c1b2f4" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43835 } } peers { permanent_uuid: "8fa70358703847a8b18809135dd09d18" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36127 } } peers { permanent_uuid: "6c77f347cb214958920c6d75fc7735c8" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 40183 } }
I20260928 16:43:38.958483 12617 consensus_queue.cc:260] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ca973be5a54d8ba5c337f1a0c1b2f4" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43835 } } peers { permanent_uuid: "8fa70358703847a8b18809135dd09d18" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36127 } } peers { permanent_uuid: "6c77f347cb214958920c6d75fc7735c8" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 40183 } }
I20260928 16:43:38.958614 12466 heartbeater.cc:503] Master 127.25.71.126:42531 was elected leader, sending a full tablet report...
I20260928 16:43:38.959125 12615 ts_tablet_manager.cc:1428] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 16:43:38.959749 12617 ts_tablet_manager.cc:1428] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260928 16:43:38.959749 12616 ts_tablet_manager.cc:1428] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260928 16:43:38.960180 12598 heartbeater.cc:503] Master 127.25.71.126:42531 was elected leader, sending a full tablet report...
I20260928 16:43:38.960180 12334 heartbeater.cc:503] Master 127.25.71.126:42531 was elected leader, sending a full tablet report...
W20260928 16:43:39.128796 12467 tablet.cc:2369] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:43:39.129818 12599 tablet.cc:2369] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:43:39.143517 12623 raft_consensus.cc:482] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:39.143862 12623 raft_consensus.cc:504] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ca973be5a54d8ba5c337f1a0c1b2f4" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43835 } } peers { permanent_uuid: "8fa70358703847a8b18809135dd09d18" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36127 } } peers { permanent_uuid: "6c77f347cb214958920c6d75fc7735c8" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 40183 } }
I20260928 16:43:39.145452 12623 leader_election.cc:288] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3ca973be5a54d8ba5c337f1a0c1b2f4 (127.25.71.67:43835), 8fa70358703847a8b18809135dd09d18 (127.25.71.66:36127)
W20260928 16:43:39.149087 12335 tablet.cc:2369] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:43:39.150804 12625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:39.145851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:36127 (local address 127.25.71.65:42425)
0928 16:43:39.146334 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:39.146348 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:39.146514 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:39.146826 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:39.146830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:39.146851 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:39.147109 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:39.147116 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:39.147941 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:39.147945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:39.149181 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:39.149194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:39.150221 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:39.150224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:39.150321 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:39.150374 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 16:43:39.150438 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 16:43:39.150529 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":88,"thread_start_us":244,"threads_started":1}
I20260928 16:43:39.150846 12614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:39.145967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:42425 (local address 127.25.71.66:36127)
0928 16:43:39.146144 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:39.146149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:39.146516 (+ 367us) server_negotiation.cc:407] Connection header received
0928 16:43:39.146662 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:39.146666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:39.146729 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:39.146795 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:39.147229 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:39.147813 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:39.149441 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:39.149938 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:39.150048 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:39.150106 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:39.150515 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 16:43:39.150684 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 16:43:39.150990 12611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:39.145967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:48953 (local address 127.25.71.67:43835)
0928 16:43:39.146198 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:39.146204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:39.146516 (+ 312us) server_negotiation.cc:407] Connection header received
0928 16:43:39.146738 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:39.146741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:39.146801 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:39.146889 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:39.147288 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:39.147843 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:39.149464 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:39.150057 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:39.150161 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:39.150207 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:39.150722 (+ 515us) server_negotiation.cc:299] Negotiation successful
0928 16:43:39.150857 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 16:43:39.151012 12624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:39.145851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:43835 (local address 127.25.71.65:48953)
0928 16:43:39.146400 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:39.146414 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:39.146564 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:39.146925 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:39.146927 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:39.146947 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:39.147158 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:39.147166 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:39.147974 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:39.147977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:39.149245 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:39.149259 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:39.150305 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:39.150308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:39.150486 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:39.150517 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 16:43:39.150599 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 16:43:39.150739 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":123,"thread_start_us":180,"threads_started":1}
I20260928 16:43:39.151988 12419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8d104faee34330a6ce226d9bcccd78" candidate_uuid: "6c77f347cb214958920c6d75fc7735c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fa70358703847a8b18809135dd09d18" is_pre_election: true
I20260928 16:43:39.151993 12551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8d104faee34330a6ce226d9bcccd78" candidate_uuid: "6c77f347cb214958920c6d75fc7735c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ca973be5a54d8ba5c337f1a0c1b2f4" is_pre_election: true
I20260928 16:43:39.152457 12551 raft_consensus.cc:2454] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c77f347cb214958920c6d75fc7735c8 in term 0.
I20260928 16:43:39.152457 12419 raft_consensus.cc:2454] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c77f347cb214958920c6d75fc7735c8 in term 0.
I20260928 16:43:39.153195 12219 leader_election.cc:302] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c77f347cb214958920c6d75fc7735c8, b3ca973be5a54d8ba5c337f1a0c1b2f4; no voters:
I20260928 16:43:39.153585 12623 raft_consensus.cc:2781] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:43:39.153677 12623 raft_consensus.cc:482] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:43:39.153712 12623 raft_consensus.cc:3037] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:39.155117 12623 raft_consensus.cc:504] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ca973be5a54d8ba5c337f1a0c1b2f4" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43835 } } peers { permanent_uuid: "8fa70358703847a8b18809135dd09d18" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36127 } } peers { permanent_uuid: "6c77f347cb214958920c6d75fc7735c8" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 40183 } }
I20260928 16:43:39.155612 12623 leader_election.cc:288] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8 [CANDIDATE]: Term 1 election: Requested vote from peers b3ca973be5a54d8ba5c337f1a0c1b2f4 (127.25.71.67:43835), 8fa70358703847a8b18809135dd09d18 (127.25.71.66:36127)
I20260928 16:43:39.156117 12551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8d104faee34330a6ce226d9bcccd78" candidate_uuid: "6c77f347cb214958920c6d75fc7735c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ca973be5a54d8ba5c337f1a0c1b2f4"
I20260928 16:43:39.156188 12419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8d104faee34330a6ce226d9bcccd78" candidate_uuid: "6c77f347cb214958920c6d75fc7735c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fa70358703847a8b18809135dd09d18"
I20260928 16:43:39.156298 12551 raft_consensus.cc:3037] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:39.156389 12419 raft_consensus.cc:3037] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:39.158141 12551 raft_consensus.cc:2454] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c77f347cb214958920c6d75fc7735c8 in term 1.
I20260928 16:43:39.158234 12419 raft_consensus.cc:2454] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c77f347cb214958920c6d75fc7735c8 in term 1.
I20260928 16:43:39.158757 12220 leader_election.cc:302] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6c77f347cb214958920c6d75fc7735c8, 8fa70358703847a8b18809135dd09d18, b3ca973be5a54d8ba5c337f1a0c1b2f4; no voters:
I20260928 16:43:39.159132 12623 raft_consensus.cc:2781] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:39.159426 12623 raft_consensus.cc:686] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8 [term 1 LEADER]: Becoming Leader. State: Replica: 6c77f347cb214958920c6d75fc7735c8, State: Running, Role: LEADER
I20260928 16:43:39.159818 12623 consensus_queue.cc:237] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ca973be5a54d8ba5c337f1a0c1b2f4" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43835 } } peers { permanent_uuid: "8fa70358703847a8b18809135dd09d18" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36127 } } peers { permanent_uuid: "6c77f347cb214958920c6d75fc7735c8" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 40183 } }
I20260928 16:43:39.164031 12142 catalog_manager.cc:5495] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c77f347cb214958920c6d75fc7735c8 (127.25.71.65). New cstate: current_term: 1 leader_uuid: "6c77f347cb214958920c6d75fc7735c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ca973be5a54d8ba5c337f1a0c1b2f4" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fa70358703847a8b18809135dd09d18" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c77f347cb214958920c6d75fc7735c8" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 40183 } health_report { overall_health: HEALTHY } } }
I20260928 16:43:39.181192 12613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:39.177066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42690 (local address 127.25.71.65:40183)
0928 16:43:39.177318 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:39.177322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:39.177334 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:43:39.177429 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:39.177432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:39.177497 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:39.177604 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:39.177905 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:39.178465 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:39.179420 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:39.179652 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:39.179744 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:39.180170 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:39.180723 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:39.180866 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:39.180963 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 16:43:39.181008 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:43:39.188970 12419 raft_consensus.cc:1261] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18 [term 1 FOLLOWER]: Refusing update from remote peer 6c77f347cb214958920c6d75fc7735c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:39.189235 12551 raft_consensus.cc:1261] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4 [term 1 FOLLOWER]: Refusing update from remote peer 6c77f347cb214958920c6d75fc7735c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:39.189973 12626 consensus_queue.cc:1035] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fa70358703847a8b18809135dd09d18" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 36127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:39.190223 12623 consensus_queue.cc:1035] T 9e8d104faee34330a6ce226d9bcccd78 P 6c77f347cb214958920c6d75fc7735c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3ca973be5a54d8ba5c337f1a0c1b2f4" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:39.200985 12631 mvcc.cc:205] Tried to move back new op lower bound from 7334354203389276160 to 7334354203281891328. Current Snapshot: MvccSnapshot[applied={T|T < 7334354203389276160}]
I20260928 16:43:39.201232 12630 mvcc.cc:205] Tried to move back new op lower bound from 7334354203389276160 to 7334354203281891328. Current Snapshot: MvccSnapshot[applied={T|T < 7334354203389276160}]
I20260928 16:43:39.213397 12141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:39.213646 12141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:39.216970 12141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52470194090741feb8547a865ef301e9: Sending DeleteTablet for 3 replicas of tablet 9e8d104faee34330a6ce226d9bcccd78
I20260928 16:43:39.218173 12531 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e8d104faee34330a6ce226d9bcccd78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:39 UTC) from {username='slave'} at 127.0.0.1:36056
I20260928 16:43:39.218235 12399 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e8d104faee34330a6ce226d9bcccd78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:39 UTC) from {username='slave'} at 127.0.0.1:44248
I20260928 16:43:39.218532 12267 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e8d104faee34330a6ce226d9bcccd78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:39 UTC) from {username='slave'} at 127.0.0.1:42676
I20260928 16:43:39.219061 12640 tablet_replica.cc:330] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18: stopping tablet replica
I20260928 16:43:39.219096 12639 tablet_replica.cc:330] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4: stopping tablet replica
I20260928 16:43:39.219462 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 12204
I20260928 16:43:39.220502 12639 raft_consensus.cc:2229] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:39.220899 12639 raft_consensus.cc:2258] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:39.221503 12640 raft_consensus.cc:2229] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:39.221932 12640 raft_consensus.cc:2258] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:39.222757 12639 ts_tablet_manager.cc:1905] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:39.223809 12640 ts_tablet_manager.cc:1905] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:39.225771 12639 ts_tablet_manager.cc:1918] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:43:39.225899 12639 log.cc:1196] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/9e8d104faee34330a6ce226d9bcccd78
I20260928 16:43:39.226202 12639 ts_tablet_manager.cc:1939] T 9e8d104faee34330a6ce226d9bcccd78 P b3ca973be5a54d8ba5c337f1a0c1b2f4: Deleting consensus metadata
I20260928 16:43:39.226812 12640 ts_tablet_manager.cc:1918] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:43:39.226939 12640 log.cc:1196] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/9e8d104faee34330a6ce226d9bcccd78
I20260928 16:43:39.227237 12640 ts_tablet_manager.cc:1939] T 9e8d104faee34330a6ce226d9bcccd78 P 8fa70358703847a8b18809135dd09d18: Deleting consensus metadata
I20260928 16:43:39.227876 12127 catalog_manager.cc:4841] TS b3ca973be5a54d8ba5c337f1a0c1b2f4 (127.25.71.67:43835): tablet 9e8d104faee34330a6ce226d9bcccd78 (table test-table [id=956202f39a534c4d8ce79e9afd0c8303]) successfully deleted
W20260928 16:43:39.228740 12130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.71.65:40183 (error 108)
I20260928 16:43:39.228902 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 12337
I20260928 16:43:39.230849 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:39.230362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:40183 (local address 127.0.0.1:42698)
0928 16:43:39.230558 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:39.230715 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.65:40183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 16:43:39.231228 12130 catalog_manager.cc:4584] TS 6c77f347cb214958920c6d75fc7735c8 (127.25.71.65:40183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e8d104faee34330a6ce226d9bcccd78: Network error: Client connection negotiation failed: client connection to 127.25.71.65:40183: connect: Connection refused (error 111)
I20260928 16:43:39.235448 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 12469
I20260928 16:43:39.238400 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:39.237922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:36127 (local address 127.0.0.1:44264)
0928 16:43:39.238164 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:39.238255 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.66:36127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 16:43:39.243144 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 12112
2026-09-28T16:43:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1385 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:39 dist-test-slave-nqgh krb5kdc[12645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:39 dist-test-slave-nqgh krb5kdc[12645](info): set up 2 sockets
Sep 28 16:43:39 dist-test-slave-nqgh krb5kdc[12645](info): commencing operation
krb5kdc: starting...
W20260928 16:43:40.220508 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.004s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:43:40 dist-test-slave-nqgh krb5kdc[12645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:43:40.359190 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:38679
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:38679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:40.496856 12658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:40.497288 12658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:40.497362 12658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:40.502216 12658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:40.502334 12658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:40.502363 12658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:40.502386 12658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:40.502409 12658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:40.509112 12658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:38679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:38679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:40.510811 12658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:40.512171 12658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:40.523051 12663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:40.523104 12666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:40.523167 12658 server_base.cc:1034] running on GCE node
W20260928 16:43:40.523104 12664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:40.524284 12658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:40.525677 12658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:40.526912 12658 hybrid_clock.cc:648] HybridClock initialized: now 1790613820526881 us; error 54 us; skew 500 ppm
Sep 28 16:43:40 dist-test-slave-nqgh krb5kdc[12645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613820, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:40.530458 12658 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:43:40.531602 12658 webserver.cc:460] Webserver started at http://127.25.71.126:36761/ using document root <none> and password file <none>
I20260928 16:43:40.532284 12658 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:40.532364 12658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:40.532596 12658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:40.534991 12658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "5096907d7d4a49bc806b998c34a88d56"
format_stamp: "Formatted at 2026-09-28 16:43:40 on dist-test-slave-nqgh"
server_key: "ef4f804a169df46117aa015f44cdacac"
server_key_iv: "10b84e9cf0c0d848e441623cd63d1555"
server_key_version: "encryptionkey@0"
I20260928 16:43:40.535638 12658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "5096907d7d4a49bc806b998c34a88d56"
format_stamp: "Formatted at 2026-09-28 16:43:40 on dist-test-slave-nqgh"
server_key: "ef4f804a169df46117aa015f44cdacac"
server_key_iv: "10b84e9cf0c0d848e441623cd63d1555"
server_key_version: "encryptionkey@0"
I20260928 16:43:40.540153 12658 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:43:40.543285 12673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:40.545408 12658 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:43:40.545620 12658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "5096907d7d4a49bc806b998c34a88d56"
format_stamp: "Formatted at 2026-09-28 16:43:40 on dist-test-slave-nqgh"
server_key: "ef4f804a169df46117aa015f44cdacac"
server_key_iv: "10b84e9cf0c0d848e441623cd63d1555"
server_key_version: "encryptionkey@0"
I20260928 16:43:40.545787 12658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:40.561600 12658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:40.565475 12658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:40.565675 12658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:40.573866 12658 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:38679
I20260928 16:43:40.573928 12726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:38679 every 8 connection(s)
I20260928 16:43:40.575042 12658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:40.578504 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 12658
I20260928 16:43:40.578667 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:43:40.578891 12727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:40.578984 25885 external_mini_cluster.cc:1428] Setting key c565aa603cb7de4b3d802b756ee78686
I20260928 16:43:40.588186 12727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56: Bootstrap starting.
Sep 28 16:43:40 dist-test-slave-nqgh krb5kdc[12645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:43:40.591771 12727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:40.592964 12727 log.cc:824] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:40.595562 12727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56: No bootstrap required, opened a new log
I20260928 16:43:40.597868 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:40.580620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.25.71.126:38679)
0928 16:43:40.581141 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:40.581159 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:40.581201 (+ 42us) server_negotiation.cc:407] Connection header received
0928 16:43:40.582526 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:40.582554 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:40.583058 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:40.583413 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:40.584648 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:40.585814 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:40.587331 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:40.587998 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:40.588186 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:40.591953 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:40.591963 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:40.591988 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:40.592042 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:40.594451 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:40.595091 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:40.595097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:40.595103 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:40.595258 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:40.595553 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:40.595556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:40.595557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:40.596134 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:40.596369 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:40.596671 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 16:43:40.597015 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":159,"threads_started":1}
I20260928 16:43:40.600342 12727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5096907d7d4a49bc806b998c34a88d56" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 38679 } }
I20260928 16:43:40.600704 12727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:40.600791 12727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5096907d7d4a49bc806b998c34a88d56, State: Initialized, Role: FOLLOWER
I20260928 16:43:40.601312 12727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5096907d7d4a49bc806b998c34a88d56" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 38679 } }
I20260928 16:43:40.601461 12727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:40.601528 12727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:40.601625 12727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:40.602984 12727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5096907d7d4a49bc806b998c34a88d56" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 38679 } }
I20260928 16:43:40.603467 12727 leader_election.cc:302] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5096907d7d4a49bc806b998c34a88d56; no voters:
I20260928 16:43:40.603943 12727 leader_election.cc:288] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:40.604179 12732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:40.604538 12732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56 [term 1 LEADER]: Becoming Leader. State: Replica: 5096907d7d4a49bc806b998c34a88d56, State: Running, Role: LEADER
I20260928 16:43:40.604956 12732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5096907d7d4a49bc806b998c34a88d56" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 38679 } }
I20260928 16:43:40.605722 12727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:40.607031 12733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5096907d7d4a49bc806b998c34a88d56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5096907d7d4a49bc806b998c34a88d56" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 38679 } } }
I20260928 16:43:40.607201 12733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:40.607510 12734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5096907d7d4a49bc806b998c34a88d56. Latest consensus state: current_term: 1 leader_uuid: "5096907d7d4a49bc806b998c34a88d56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5096907d7d4a49bc806b998c34a88d56" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 38679 } } }
I20260928 16:43:40.607626 12734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56 [sys.catalog]: This master's current role is: LEADER
W20260928 16:43:40.611744 12747 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:43:40.611874 12747 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:43:40.611965 12741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:40.612898 12741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:40.620702 12741 catalog_manager.cc:1347] Generated new cluster ID: 8714ce760b2d428b9f89f24ca274c39e
I20260928 16:43:40.620828 12741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:40.637954 12741 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:40.639562 12741 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:40.648031 12741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56: Generated new TSK 0
I20260928 16:43:40.649286 12741 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
Entry for principal kudu/127.25.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab.
I20260928 16:43:40.695132 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:38679
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:40.832798 12753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:40.833218 12753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:40.833274 12753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:40.838158 12753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:40.838286 12753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:40.838361 12753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:43:40.845162 12753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.65.keytab
--principal=kudu/127.25.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:38679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:40.846863 12753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:40.848299 12753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:40.861100 12759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:40.861086 12758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:40.861086 12761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:40.861589 12753 server_base.cc:1034] running on GCE node
I20260928 16:43:40.862221 12753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:40.863363 12753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:40.864616 12753 hybrid_clock.cc:648] HybridClock initialized: now 1790613820864571 us; error 63 us; skew 500 ppm
Sep 28 16:43:40 dist-test-slave-nqgh krb5kdc[12645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613820, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:40.868446 12753 init.cc:377] Logged in from keytab as kudu/127.25.71.65@KRBTEST.COM (short username kudu)
I20260928 16:43:40.869761 12753 webserver.cc:460] Webserver started at http://127.25.71.65:45467/ using document root <none> and password file <none>
I20260928 16:43:40.870520 12753 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:40.870607 12753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:40.870908 12753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:40.873688 12753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "e476f36ab99946808e2afc5681b0b721"
format_stamp: "Formatted at 2026-09-28 16:43:40 on dist-test-slave-nqgh"
server_key: "1bf1843766eb182f571c9a947e273b36"
server_key_iv: "41673250364ea07b733338bdca1b8999"
server_key_version: "encryptionkey@0"
I20260928 16:43:40.874368 12753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "e476f36ab99946808e2afc5681b0b721"
format_stamp: "Formatted at 2026-09-28 16:43:40 on dist-test-slave-nqgh"
server_key: "1bf1843766eb182f571c9a947e273b36"
server_key_iv: "41673250364ea07b733338bdca1b8999"
server_key_version: "encryptionkey@0"
I20260928 16:43:40.879562 12753 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 16:43:40.883416 12768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:40.885468 12753 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:43:40.885659 12753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "e476f36ab99946808e2afc5681b0b721"
format_stamp: "Formatted at 2026-09-28 16:43:40 on dist-test-slave-nqgh"
server_key: "1bf1843766eb182f571c9a947e273b36"
server_key_iv: "41673250364ea07b733338bdca1b8999"
server_key_version: "encryptionkey@0"
I20260928 16:43:40.885841 12753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:40.923585 12753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:40.927994 12753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:40.928263 12753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:40.929051 12753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:40.930198 12753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:40.930255 12753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:40.930305 12753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:40.930320 12753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:40.941533 12753 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:44171
I20260928 16:43:40.941658 12881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:44171 every 8 connection(s)
I20260928 16:43:40.942793 12753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:43:40.945258 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 12753
I20260928 16:43:40.945431 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:43:40.945768 25885 external_mini_cluster.cc:1428] Setting key 31dbae1d4cc132057d36b0be540d111c
Sep 28 16:43:40 dist-test-slave-nqgh krb5kdc[12645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613820, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.65@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.66@KRBTEST.COM" created.
I20260928 16:43:40.962669 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:40.945430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:34143 (local address 127.25.71.126:38679)
0928 16:43:40.945609 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:40.945614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:40.947163 (+ 1549us) server_negotiation.cc:407] Connection header received
0928 16:43:40.948355 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:40.948359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:40.948427 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:40.948505 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:40.950185 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:40.950934 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:40.952995 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:40.953292 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:40.953422 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:40.957413 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:40.957418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:40.957421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:40.957463 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:40.959212 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:40.960085 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:40.960089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:40.960091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:40.960291 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:40.960833 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:40.960836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:40.960837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:40.961097 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:40.961264 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:40.962112 (+ 848us) server_negotiation.cc:299] Negotiation successful
0928 16:43:40.962407 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 16:43:40.963464 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:40.945926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:38679 (local address 127.25.71.65:34143)
0928 16:43:40.946939 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:40.947005 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:40.948092 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:40.948732 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:40.948742 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:40.949277 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:40.949958 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:40.949976 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:40.951132 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:40.951136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:40.952769 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:40.952788 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:40.953545 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:40.953550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:40.953846 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:40.954740 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:40.954770 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:40.957091 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:40.959437 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:40.959445 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:40.959464 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:40.959923 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:40.960439 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:40.960443 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:40.960445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:40.960703 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:40.961280 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:40.961288 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:40.961713 (+ 425us) client_negotiation.cc:769] Sending connection context
0928 16:43:40.961985 (+ 272us) client_negotiation.cc:240] Negotiation successful
0928 16:43:40.962344 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":187,"threads_started":1}
I20260928 16:43:40.965194 12882 heartbeater.cc:347] Connected to a master server at 127.25.71.126:38679
I20260928 16:43:40.965564 12882 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:40.966166 12882 heartbeater.cc:511] Master 127.25.71.126:38679 requested a full tablet report, sending...
I20260928 16:43:40.968618 12688 ts_manager.cc:194] Registered new tserver with Master: e476f36ab99946808e2afc5681b0b721 (127.25.71.65:44171)
I20260928 16:43:40.970551 12688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.65@KRBTEST.COM'} at 127.25.71.65:34143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
Entry for principal kudu/127.25.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab.
I20260928 16:43:40.983697 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:38679
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:41.128847 12887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:41.129354 12887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:41.129427 12887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:41.135200 12887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:41.135322 12887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:41.135397 12887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:43:41.142275 12887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.66.keytab
--principal=kudu/127.25.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:38679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:41.143868 12887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:41.145190 12887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:41.158954 12892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:41.158974 12895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:41.158985 12893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:41.159360 12887 server_base.cc:1034] running on GCE node
I20260928 16:43:41.160005 12887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:41.160923 12887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:41.162186 12887 hybrid_clock.cc:648] HybridClock initialized: now 1790613821162137 us; error 64 us; skew 500 ppm
Sep 28 16:43:41 dist-test-slave-nqgh krb5kdc[12645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613821, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:41.166060 12887 init.cc:377] Logged in from keytab as kudu/127.25.71.66@KRBTEST.COM (short username kudu)
I20260928 16:43:41.167533 12887 webserver.cc:460] Webserver started at http://127.25.71.66:43959/ using document root <none> and password file <none>
I20260928 16:43:41.168277 12887 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:41.168351 12887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:41.168597 12887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:41.171355 12887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "0fbc34a19aed4418b5cf9830be6b6254"
format_stamp: "Formatted at 2026-09-28 16:43:41 on dist-test-slave-nqgh"
server_key: "6a3abb8b34aa330747486d4453051055"
server_key_iv: "d9b1ce9b74cef64e537923f410746304"
server_key_version: "encryptionkey@0"
I20260928 16:43:41.172098 12887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "0fbc34a19aed4418b5cf9830be6b6254"
format_stamp: "Formatted at 2026-09-28 16:43:41 on dist-test-slave-nqgh"
server_key: "6a3abb8b34aa330747486d4453051055"
server_key_iv: "d9b1ce9b74cef64e537923f410746304"
server_key_version: "encryptionkey@0"
I20260928 16:43:41.177182 12887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:43:41.180872 12902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:41.182785 12887 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:43:41.182966 12887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "0fbc34a19aed4418b5cf9830be6b6254"
format_stamp: "Formatted at 2026-09-28 16:43:41 on dist-test-slave-nqgh"
server_key: "6a3abb8b34aa330747486d4453051055"
server_key_iv: "d9b1ce9b74cef64e537923f410746304"
server_key_version: "encryptionkey@0"
I20260928 16:43:41.183214 12887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:41.203140 12887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:41.207156 12887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:41.207381 12887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:41.208175 12887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:41.209257 12887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:41.209318 12887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:41.209385 12887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:41.209414 12887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:41.221369 12887 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:38921
I20260928 16:43:41.221421 13016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:38921 every 8 connection(s)
I20260928 16:43:41.222805 12887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:43:41.233225 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 12887
I20260928 16:43:41.233414 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:43:41.233814 25885 external_mini_cluster.cc:1428] Setting key 401091a11e80192d6d62476e792f3a7f
Sep 28 16:43:41 dist-test-slave-nqgh krb5kdc[12645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613821, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.66@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:43:41.240316 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.225427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:39979 (local address 127.25.71.126:38679)
0928 16:43:41.225575 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:41.225579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:41.227127 (+ 1548us) server_negotiation.cc:407] Connection header received
0928 16:43:41.228166 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:41.228170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:41.228244 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:41.228359 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:41.230077 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.230822 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.232253 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.232532 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.232658 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:41.236213 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:41.236217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:41.236219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:41.236250 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:41.237714 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:41.238410 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:41.238413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:41.238415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:41.238498 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:41.238929 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:41.238932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:41.238934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:41.239180 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:41.239276 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:41.239948 (+ 672us) server_negotiation.cc:299] Negotiation successful
0928 16:43:41.240122 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:43:41.241460 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.225955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:38679 (local address 127.25.71.66:39979)
0928 16:43:41.226908 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:41.226968 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:41.227891 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:41.228729 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:41.228739 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:41.229250 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:41.229823 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:41.229838 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.231022 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.231025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:41.232077 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:41.232090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.232730 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.232733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:41.232926 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:41.233805 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:41.233826 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:41.236007 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:41.237868 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:41.237878 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:41.237902 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:41.238293 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:41.238597 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:41.238600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:41.238603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:41.238830 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:41.239292 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:41.239301 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:41.239664 (+ 363us) client_negotiation.cc:769] Sending connection context
0928 16:43:41.239870 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 16:43:41.240334 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":176,"threads_started":1}
I20260928 16:43:41.243080 13017 heartbeater.cc:347] Connected to a master server at 127.25.71.126:38679
I20260928 16:43:41.243544 13017 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.25.71.67@KRBTEST.COM; defaulting to no policy
I20260928 16:43:41.244352 13017 heartbeater.cc:511] Master 127.25.71.126:38679 requested a full tablet report, sending...
I20260928 16:43:41.245976 12688 ts_manager.cc:194] Registered new tserver with Master: 0fbc34a19aed4418b5cf9830be6b6254 (127.25.71.66:38921)
I20260928 16:43:41.246671 12688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.66@KRBTEST.COM'} at 127.25.71.66:39979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
Entry for principal kudu/127.25.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab.
I20260928 16:43:41.268059 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:38679
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:41.393896 13022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:41.394203 13022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:41.394249 13022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:41.398500 13022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:41.398598 13022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:41.398715 13022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:43:41.404489 13022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.67.keytab
--principal=kudu/127.25.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:38679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:41.405684 13022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:41.406929 13022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:41.418262 13028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:41.418385 13027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:41.418262 13030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:41.418555 13022 server_base.cc:1034] running on GCE node
I20260928 16:43:41.419275 13022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:41.420037 13022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:41.421273 13022 hybrid_clock.cc:648] HybridClock initialized: now 1790613821421251 us; error 48 us; skew 500 ppm
Sep 28 16:43:41 dist-test-slave-nqgh krb5kdc[12645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613821, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:41.425171 13022 init.cc:377] Logged in from keytab as kudu/127.25.71.67@KRBTEST.COM (short username kudu)
I20260928 16:43:41.426401 13022 webserver.cc:460] Webserver started at http://127.25.71.67:40913/ using document root <none> and password file <none>
I20260928 16:43:41.427117 13022 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:41.427196 13022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:41.427423 13022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:41.430680 13022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "bd71aaa953574ceea0b820af3101112d"
format_stamp: "Formatted at 2026-09-28 16:43:41 on dist-test-slave-nqgh"
server_key: "22f422cc0ac837cfad9a8cecfd7aa817"
server_key_iv: "12866eff4939fea272314cbd7e225feb"
server_key_version: "encryptionkey@0"
I20260928 16:43:41.431504 13022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "bd71aaa953574ceea0b820af3101112d"
format_stamp: "Formatted at 2026-09-28 16:43:41 on dist-test-slave-nqgh"
server_key: "22f422cc0ac837cfad9a8cecfd7aa817"
server_key_iv: "12866eff4939fea272314cbd7e225feb"
server_key_version: "encryptionkey@0"
I20260928 16:43:41.436385 13022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 16:43:41.439906 13037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:41.441627 13022 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:43:41.441810 13022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "bd71aaa953574ceea0b820af3101112d"
format_stamp: "Formatted at 2026-09-28 16:43:41 on dist-test-slave-nqgh"
server_key: "22f422cc0ac837cfad9a8cecfd7aa817"
server_key_iv: "12866eff4939fea272314cbd7e225feb"
server_key_version: "encryptionkey@0"
I20260928 16:43:41.442039 13022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:41.453953 13022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:41.457826 13022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:41.458041 13022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:41.458781 13022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:41.459913 13022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:41.459972 13022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:41.460043 13022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:41.460073 13022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:41.470649 13022 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:43861
I20260928 16:43:41.470736 13150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:43861 every 8 connection(s)
I20260928 16:43:41.471719 13022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:43:41.477645 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 13022
I20260928 16:43:41.477828 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:43:41.478200 25885 external_mini_cluster.cc:1428] Setting key 08de08e620e21de587b0a6c6d750823d
Sep 28 16:43:41 dist-test-slave-nqgh krb5kdc[12645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613821, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.67@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:43:41.489034 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.474383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:60395 (local address 127.25.71.126:38679)
0928 16:43:41.474580 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:41.474587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:41.476050 (+ 1463us) server_negotiation.cc:407] Connection header received
0928 16:43:41.477325 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:41.477328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:41.477404 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:41.477517 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:41.479489 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.480131 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.481487 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.481726 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.481820 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:41.484953 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:41.484956 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:41.484959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:41.484991 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:41.486426 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:41.487065 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:41.487069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:41.487070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:41.487157 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:41.487518 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:41.487520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:41.487522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:41.487755 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:41.487845 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:41.488643 (+ 798us) server_negotiation.cc:299] Negotiation successful
0928 16:43:41.488817 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:43:41.490319 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.474743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:38679 (local address 127.25.71.67:60395)
0928 16:43:41.475779 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:41.475858 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:41.476890 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:41.477826 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:41.477838 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:41.478498 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:41.479283 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:41.479303 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.480289 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.480293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:41.481350 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:41.481362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.481897 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.481901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:41.482106 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:41.482847 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:41.482866 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:41.484726 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:41.486560 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:41.486567 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:41.486591 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:41.486951 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:41.487244 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:41.487247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:41.487250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:41.487418 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:41.487862 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:41.487870 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:41.488256 (+ 386us) client_negotiation.cc:769] Sending connection context
0928 16:43:41.488609 (+ 353us) client_negotiation.cc:240] Negotiation successful
0928 16:43:41.489082 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":140,"threads_started":1}
I20260928 16:43:41.492173 13151 heartbeater.cc:347] Connected to a master server at 127.25.71.126:38679
I20260928 16:43:41.492496 13151 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:41.493261 13151 heartbeater.cc:511] Master 127.25.71.126:38679 requested a full tablet report, sending...
I20260928 16:43:41.494689 12688 ts_manager.cc:194] Registered new tserver with Master: bd71aaa953574ceea0b820af3101112d (127.25.71.67:43861)
I20260928 16:43:41.495424 12688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.71.67@KRBTEST.COM'} at 127.25.71.67:60395
I20260928 16:43:41.505834 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:43:41.524493 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.508421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55244 (local address 127.25.71.126:38679)
0928 16:43:41.508620 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:41.508625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:41.508852 (+ 227us) server_negotiation.cc:407] Connection header received
0928 16:43:41.509040 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:41.509043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:41.509092 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:41.509164 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:41.510226 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.510878 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.511792 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.512018 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.512107 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:41.519868 (+ 7761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:41.519872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:41.519875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:41.519913 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:41.521948 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:41.523082 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:41.523087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:41.523088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:41.523200 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:41.523492 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:41.523495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:41.523496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:41.523769 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:41.523903 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:41.524166 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 16:43:41.524307 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 16:43:41.529045 12688 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:43:41.533357 12688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:43:41.553480 13163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.545316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:43861 (local address 127.0.0.1:58916)
0928 16:43:41.545923 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:41.545940 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:41.546093 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:41.547606 (+ 1513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:41.547609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:41.547632 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:41.547890 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:41.547899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.549574 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.549577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:41.551375 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:41.551387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.552746 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.552749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:41.552884 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:41.552927 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 16:43:41.553058 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 16:43:41.553287 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":504,"thread_start_us":178,"threads_started":1}
I20260928 16:43:41.553499 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.545078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:38921 (local address 127.0.0.1:60916)
0928 16:43:41.545666 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:41.545720 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:41.545875 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:41.547675 (+ 1800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:41.547678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:41.547690 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:41.547890 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:41.547900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.551074 (+ 3174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.551078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:41.551885 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:41.551896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.552913 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.552917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:41.553023 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:41.553060 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 16:43:41.553115 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 16:43:41.553287 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":125,"thread_start_us":92,"threads_started":1}
I20260928 16:43:41.554351 13165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.545986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:44171 (local address 127.0.0.1:34420)
0928 16:43:41.546304 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:41.546318 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:41.546771 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:41.547464 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:41.547471 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:41.547498 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:41.547803 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:41.547813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.549572 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.549576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:41.552315 (+ 2739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:41.552327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.553950 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.553954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:41.554045 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:41.554068 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:43:41.554138 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 16:43:41.554227 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":1251,"thread_start_us":119,"threads_started":1}
I20260928 16:43:41.554464 13164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.545322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60916 (local address 127.25.71.66:38921)
0928 16:43:41.546521 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:41.546537 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:41.547018 (+ 481us) server_negotiation.cc:407] Connection header received
0928 16:43:41.547151 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:41.547159 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:41.547493 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:41.547702 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:41.549432 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.550936 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.552023 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.552738 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.554012 (+ 1274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:41.554158 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:41.554216 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 16:43:41.554318 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1072,"thread_start_us":181,"threads_started":1}
I20260928 16:43:41.554378 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.546566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58916 (local address 127.25.71.67:43861)
0928 16:43:41.547010 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:41.547021 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:41.547041 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:43:41.547113 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:41.547117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:41.547259 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:41.547422 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:41.548063 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.549374 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.551532 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.552603 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.552712 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:41.552923 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:41.553122 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 16:43:41.553271 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":125,"threads_started":1}
I20260928 16:43:41.555603 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.546347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34420 (local address 127.25.71.65:44171)
0928 16:43:41.546887 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:41.546895 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:41.546925 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:43:41.547016 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:41.547022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:41.547251 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:41.547422 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:41.547974 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.549407 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.552501 (+ 3094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.553596 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.553708 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:41.553878 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:41.554689 (+ 811us) server_negotiation.cc:299] Negotiation successful
0928 16:43:41.555462 (+ 773us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":89,"threads_started":1}
I20260928 16:43:41.558302 12816 tablet_service.cc:1467] Processing CreateTablet for tablet e25e22ef9fd947059f0aba837e747552 (DEFAULT_TABLE table=test-table [id=b54ac0bdf40a445cbd8a212c238fc718]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:41.558478 13085 tablet_service.cc:1467] Processing CreateTablet for tablet e25e22ef9fd947059f0aba837e747552 (DEFAULT_TABLE table=test-table [id=b54ac0bdf40a445cbd8a212c238fc718]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:41.558478 12950 tablet_service.cc:1467] Processing CreateTablet for tablet e25e22ef9fd947059f0aba837e747552 (DEFAULT_TABLE table=test-table [id=b54ac0bdf40a445cbd8a212c238fc718]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:41.559477 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25e22ef9fd947059f0aba837e747552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:41.559917 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25e22ef9fd947059f0aba837e747552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:41.559918 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25e22ef9fd947059f0aba837e747552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:41.566490 13168 tablet_bootstrap.cc:492] T e25e22ef9fd947059f0aba837e747552 P e476f36ab99946808e2afc5681b0b721: Bootstrap starting.
I20260928 16:43:41.570314 13168 tablet_bootstrap.cc:654] T e25e22ef9fd947059f0aba837e747552 P e476f36ab99946808e2afc5681b0b721: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:41.570935 13170 tablet_bootstrap.cc:492] T e25e22ef9fd947059f0aba837e747552 P bd71aaa953574ceea0b820af3101112d: Bootstrap starting.
I20260928 16:43:41.571421 13168 log.cc:824] T e25e22ef9fd947059f0aba837e747552 P e476f36ab99946808e2afc5681b0b721: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:41.572034 13169 tablet_bootstrap.cc:492] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254: Bootstrap starting.
I20260928 16:43:41.573540 13170 tablet_bootstrap.cc:654] T e25e22ef9fd947059f0aba837e747552 P bd71aaa953574ceea0b820af3101112d: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:41.574210 13169 tablet_bootstrap.cc:654] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:41.574505 13168 tablet_bootstrap.cc:492] T e25e22ef9fd947059f0aba837e747552 P e476f36ab99946808e2afc5681b0b721: No bootstrap required, opened a new log
I20260928 16:43:41.574877 13168 ts_tablet_manager.cc:1397] T e25e22ef9fd947059f0aba837e747552 P e476f36ab99946808e2afc5681b0b721: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260928 16:43:41.575003 13170 log.cc:824] T e25e22ef9fd947059f0aba837e747552 P bd71aaa953574ceea0b820af3101112d: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:41.575445 13169 log.cc:824] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:41.577796 13170 tablet_bootstrap.cc:492] T e25e22ef9fd947059f0aba837e747552 P bd71aaa953574ceea0b820af3101112d: No bootstrap required, opened a new log
I20260928 16:43:41.578078 13170 ts_tablet_manager.cc:1397] T e25e22ef9fd947059f0aba837e747552 P bd71aaa953574ceea0b820af3101112d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 16:43:41.579384 13169 tablet_bootstrap.cc:492] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254: No bootstrap required, opened a new log
I20260928 16:43:41.579624 13169 ts_tablet_manager.cc:1397] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260928 16:43:41.580902 13168 raft_consensus.cc:348] T e25e22ef9fd947059f0aba837e747552 P e476f36ab99946808e2afc5681b0b721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbc34a19aed4418b5cf9830be6b6254" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 38921 } } peers { permanent_uuid: "bd71aaa953574ceea0b820af3101112d" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43861 } } peers { permanent_uuid: "e476f36ab99946808e2afc5681b0b721" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 44171 } }
I20260928 16:43:41.581451 13168 raft_consensus.cc:374] T e25e22ef9fd947059f0aba837e747552 P e476f36ab99946808e2afc5681b0b721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:41.581547 13168 raft_consensus.cc:729] T e25e22ef9fd947059f0aba837e747552 P e476f36ab99946808e2afc5681b0b721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e476f36ab99946808e2afc5681b0b721, State: Initialized, Role: FOLLOWER
I20260928 16:43:41.582288 13168 consensus_queue.cc:260] T e25e22ef9fd947059f0aba837e747552 P e476f36ab99946808e2afc5681b0b721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbc34a19aed4418b5cf9830be6b6254" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 38921 } } peers { permanent_uuid: "bd71aaa953574ceea0b820af3101112d" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43861 } } peers { permanent_uuid: "e476f36ab99946808e2afc5681b0b721" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 44171 } }
I20260928 16:43:41.582439 13170 raft_consensus.cc:348] T e25e22ef9fd947059f0aba837e747552 P bd71aaa953574ceea0b820af3101112d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbc34a19aed4418b5cf9830be6b6254" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 38921 } } peers { permanent_uuid: "bd71aaa953574ceea0b820af3101112d" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43861 } } peers { permanent_uuid: "e476f36ab99946808e2afc5681b0b721" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 44171 } }
I20260928 16:43:41.582903 13170 raft_consensus.cc:374] T e25e22ef9fd947059f0aba837e747552 P bd71aaa953574ceea0b820af3101112d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:41.582978 13170 raft_consensus.cc:729] T e25e22ef9fd947059f0aba837e747552 P bd71aaa953574ceea0b820af3101112d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd71aaa953574ceea0b820af3101112d, State: Initialized, Role: FOLLOWER
I20260928 16:43:41.584034 13168 ts_tablet_manager.cc:1428] T e25e22ef9fd947059f0aba837e747552 P e476f36ab99946808e2afc5681b0b721: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260928 16:43:41.583591 13170 consensus_queue.cc:260] T e25e22ef9fd947059f0aba837e747552 P bd71aaa953574ceea0b820af3101112d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbc34a19aed4418b5cf9830be6b6254" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 38921 } } peers { permanent_uuid: "bd71aaa953574ceea0b820af3101112d" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43861 } } peers { permanent_uuid: "e476f36ab99946808e2afc5681b0b721" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 44171 } }
I20260928 16:43:41.584036 13169 raft_consensus.cc:348] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbc34a19aed4418b5cf9830be6b6254" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 38921 } } peers { permanent_uuid: "bd71aaa953574ceea0b820af3101112d" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43861 } } peers { permanent_uuid: "e476f36ab99946808e2afc5681b0b721" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 44171 } }
I20260928 16:43:41.584385 13169 raft_consensus.cc:374] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:41.584476 13169 raft_consensus.cc:729] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbc34a19aed4418b5cf9830be6b6254, State: Initialized, Role: FOLLOWER
I20260928 16:43:41.584736 12882 heartbeater.cc:503] Master 127.25.71.126:38679 was elected leader, sending a full tablet report...
I20260928 16:43:41.585302 13170 ts_tablet_manager.cc:1428] T e25e22ef9fd947059f0aba837e747552 P bd71aaa953574ceea0b820af3101112d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 16:43:41.585971 13151 heartbeater.cc:503] Master 127.25.71.126:38679 was elected leader, sending a full tablet report...
I20260928 16:43:41.587678 13169 consensus_queue.cc:260] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbc34a19aed4418b5cf9830be6b6254" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 38921 } } peers { permanent_uuid: "bd71aaa953574ceea0b820af3101112d" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43861 } } peers { permanent_uuid: "e476f36ab99946808e2afc5681b0b721" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 44171 } }
I20260928 16:43:41.588821 13017 heartbeater.cc:503] Master 127.25.71.126:38679 was elected leader, sending a full tablet report...
I20260928 16:43:41.589277 13169 ts_tablet_manager.cc:1428] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260928 16:43:41.595330 13176 raft_consensus.cc:482] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:41.595620 13176 raft_consensus.cc:504] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbc34a19aed4418b5cf9830be6b6254" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 38921 } } peers { permanent_uuid: "bd71aaa953574ceea0b820af3101112d" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43861 } } peers { permanent_uuid: "e476f36ab99946808e2afc5681b0b721" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 44171 } }
I20260928 16:43:41.597612 13176 leader_election.cc:288] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd71aaa953574ceea0b820af3101112d (127.25.71.67:43861), e476f36ab99946808e2afc5681b0b721 (127.25.71.65:44171)
I20260928 16:43:41.601476 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.597435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:44171 (local address 127.25.71.66:58691)
0928 16:43:41.597647 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:41.597665 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:41.597777 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:41.598329 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:41.598333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:41.598358 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:41.598484 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:41.598491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.599346 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.599350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:41.600282 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:41.600294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.601009 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.601013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:41.601129 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:41.601152 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:43:41.601220 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 16:43:41.601329 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":49}
I20260928 16:43:41.601476 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.597980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:58691 (local address 127.25.71.65:44171)
0928 16:43:41.598094 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:41.598098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:41.598109 (+ 11us) server_negotiation.cc:407] Connection header received
0928 16:43:41.598171 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:41.598173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:41.598212 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:41.598281 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:41.598607 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.599198 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.600451 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.600896 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.601009 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:41.601068 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:41.601249 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 16:43:41.601329 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 16:43:41.601693 13177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.597435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:43861 (local address 127.25.71.66:34567)
0928 16:43:41.598135 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:41.598148 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:41.598294 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:41.598553 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:41.598555 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:41.598575 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:41.598751 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:41.598757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.599544 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.599548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:41.600463 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:41.600473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.601189 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.601192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:41.601318 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:41.601344 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:43:41.601412 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 16:43:41.601509 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":290,"thread_start_us":164,"threads_started":1}
I20260928 16:43:41.601682 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.597620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:34567 (local address 127.25.71.67:43861)
0928 16:43:41.597785 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:41.597790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:41.598258 (+ 468us) server_negotiation.cc:407] Connection header received
0928 16:43:41.598402 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:41.598405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:41.598457 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:41.598523 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:41.598864 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.599422 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.600600 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.601064 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.601189 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:41.601257 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:41.601442 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 16:43:41.601538 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:43:41.602479 12836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25e22ef9fd947059f0aba837e747552" candidate_uuid: "0fbc34a19aed4418b5cf9830be6b6254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e476f36ab99946808e2afc5681b0b721" is_pre_election: true
I20260928 16:43:41.602496 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25e22ef9fd947059f0aba837e747552" candidate_uuid: "0fbc34a19aed4418b5cf9830be6b6254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd71aaa953574ceea0b820af3101112d" is_pre_election: true
I20260928 16:43:41.602885 13105 raft_consensus.cc:2454] T e25e22ef9fd947059f0aba837e747552 P bd71aaa953574ceea0b820af3101112d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbc34a19aed4418b5cf9830be6b6254 in term 0.
I20260928 16:43:41.602977 12836 raft_consensus.cc:2454] T e25e22ef9fd947059f0aba837e747552 P e476f36ab99946808e2afc5681b0b721 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbc34a19aed4418b5cf9830be6b6254 in term 0.
I20260928 16:43:41.603499 12904 leader_election.cc:302] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbc34a19aed4418b5cf9830be6b6254, bd71aaa953574ceea0b820af3101112d; no voters:
I20260928 16:43:41.603864 13176 raft_consensus.cc:2781] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:43:41.603969 13176 raft_consensus.cc:482] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:43:41.604012 13176 raft_consensus.cc:3037] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:41.605419 13176 raft_consensus.cc:504] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbc34a19aed4418b5cf9830be6b6254" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 38921 } } peers { permanent_uuid: "bd71aaa953574ceea0b820af3101112d" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43861 } } peers { permanent_uuid: "e476f36ab99946808e2afc5681b0b721" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 44171 } }
I20260928 16:43:41.605916 13176 leader_election.cc:288] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 [CANDIDATE]: Term 1 election: Requested vote from peers bd71aaa953574ceea0b820af3101112d (127.25.71.67:43861), e476f36ab99946808e2afc5681b0b721 (127.25.71.65:44171)
I20260928 16:43:41.606546 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25e22ef9fd947059f0aba837e747552" candidate_uuid: "0fbc34a19aed4418b5cf9830be6b6254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd71aaa953574ceea0b820af3101112d"
I20260928 16:43:41.606746 12836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25e22ef9fd947059f0aba837e747552" candidate_uuid: "0fbc34a19aed4418b5cf9830be6b6254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e476f36ab99946808e2afc5681b0b721"
I20260928 16:43:41.606805 13105 raft_consensus.cc:3037] T e25e22ef9fd947059f0aba837e747552 P bd71aaa953574ceea0b820af3101112d [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:41.606942 12836 raft_consensus.cc:3037] T e25e22ef9fd947059f0aba837e747552 P e476f36ab99946808e2afc5681b0b721 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:41.608304 13105 raft_consensus.cc:2454] T e25e22ef9fd947059f0aba837e747552 P bd71aaa953574ceea0b820af3101112d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fbc34a19aed4418b5cf9830be6b6254 in term 1.
I20260928 16:43:41.608523 12836 raft_consensus.cc:2454] T e25e22ef9fd947059f0aba837e747552 P e476f36ab99946808e2afc5681b0b721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fbc34a19aed4418b5cf9830be6b6254 in term 1.
I20260928 16:43:41.608768 12904 leader_election.cc:302] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbc34a19aed4418b5cf9830be6b6254, bd71aaa953574ceea0b820af3101112d; no voters:
I20260928 16:43:41.609030 13176 raft_consensus.cc:2781] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:41.609315 13176 raft_consensus.cc:686] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 [term 1 LEADER]: Becoming Leader. State: Replica: 0fbc34a19aed4418b5cf9830be6b6254, State: Running, Role: LEADER
I20260928 16:43:41.609727 13176 consensus_queue.cc:237] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbc34a19aed4418b5cf9830be6b6254" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 38921 } } peers { permanent_uuid: "bd71aaa953574ceea0b820af3101112d" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43861 } } peers { permanent_uuid: "e476f36ab99946808e2afc5681b0b721" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 44171 } }
I20260928 16:43:41.613754 12689 catalog_manager.cc:5495] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fbc34a19aed4418b5cf9830be6b6254 (127.25.71.66). New cstate: current_term: 1 leader_uuid: "0fbc34a19aed4418b5cf9830be6b6254" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbc34a19aed4418b5cf9830be6b6254" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 38921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd71aaa953574ceea0b820af3101112d" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e476f36ab99946808e2afc5681b0b721" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 44171 } health_report { overall_health: UNKNOWN } } }
I20260928 16:43:41.638695 13164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.633544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60932 (local address 127.25.71.66:38921)
0928 16:43:41.633881 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:41.633888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:41.633911 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:43:41.634055 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:41.634059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:41.634161 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:41.634318 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:41.634765 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.635457 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.636559 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.636816 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.636919 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:41.637390 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:41.638039 (+ 649us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:41.638159 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:41.638305 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 16:43:41.638408 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 16:43:41.646960 12836 raft_consensus.cc:1261] T e25e22ef9fd947059f0aba837e747552 P e476f36ab99946808e2afc5681b0b721 [term 1 FOLLOWER]: Refusing update from remote peer 0fbc34a19aed4418b5cf9830be6b6254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:41.647079 13105 raft_consensus.cc:1261] T e25e22ef9fd947059f0aba837e747552 P bd71aaa953574ceea0b820af3101112d [term 1 FOLLOWER]: Refusing update from remote peer 0fbc34a19aed4418b5cf9830be6b6254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:41.648228 13176 consensus_queue.cc:1035] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd71aaa953574ceea0b820af3101112d" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:41.648591 13178 consensus_queue.cc:1035] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e476f36ab99946808e2afc5681b0b721" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 44171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:41.660434 13182 mvcc.cc:205] Tried to move back new op lower bound from 7334354213455204352 to 7334354213317009408. Current Snapshot: MvccSnapshot[applied={T|T < 7334354213455204352}]
I20260928 16:43:41.671630 12688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:41.671818 12688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:41.675312 12688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5096907d7d4a49bc806b998c34a88d56: Sending DeleteTablet for 3 replicas of tablet e25e22ef9fd947059f0aba837e747552
I20260928 16:43:41.676383 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet e25e22ef9fd947059f0aba837e747552 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:41 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:34420
I20260928 16:43:41.676461 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet e25e22ef9fd947059f0aba837e747552 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:41 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:60916
I20260928 16:43:41.676468 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet e25e22ef9fd947059f0aba837e747552 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:41 UTC) from {username='kudu', principal='kudu/127.25.71.126@KRBTEST.COM'} at 127.0.0.1:58916
I20260928 16:43:41.677278 13190 tablet_replica.cc:330] T e25e22ef9fd947059f0aba837e747552 P e476f36ab99946808e2afc5681b0b721: stopping tablet replica
I20260928 16:43:41.677323 13192 tablet_replica.cc:330] T e25e22ef9fd947059f0aba837e747552 P bd71aaa953574ceea0b820af3101112d: stopping tablet replica
I20260928 16:43:41.677664 13191 tablet_replica.cc:330] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254: stopping tablet replica
I20260928 16:43:41.677685 13190 raft_consensus.cc:2229] T e25e22ef9fd947059f0aba837e747552 P e476f36ab99946808e2afc5681b0b721 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:41.678063 13191 raft_consensus.cc:2229] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:43:41.678076 13190 raft_consensus.cc:2258] T e25e22ef9fd947059f0aba837e747552 P e476f36ab99946808e2afc5681b0b721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:41.678614 13192 raft_consensus.cc:2229] T e25e22ef9fd947059f0aba837e747552 P bd71aaa953574ceea0b820af3101112d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:41.678659 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 12753
I20260928 16:43:41.679087 13191 raft_consensus.cc:2258] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:41.679987 13192 raft_consensus.cc:2258] T e25e22ef9fd947059f0aba837e747552 P bd71aaa953574ceea0b820af3101112d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:41.681789 13191 ts_tablet_manager.cc:1905] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:41.683713 13192 ts_tablet_manager.cc:1905] T e25e22ef9fd947059f0aba837e747552 P bd71aaa953574ceea0b820af3101112d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:41.686717 13192 ts_tablet_manager.cc:1918] T e25e22ef9fd947059f0aba837e747552 P bd71aaa953574ceea0b820af3101112d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:43:41.686851 13192 log.cc:1196] T e25e22ef9fd947059f0aba837e747552 P bd71aaa953574ceea0b820af3101112d: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/e25e22ef9fd947059f0aba837e747552
I20260928 16:43:41.687207 13192 ts_tablet_manager.cc:1939] T e25e22ef9fd947059f0aba837e747552 P bd71aaa953574ceea0b820af3101112d: Deleting consensus metadata
I20260928 16:43:41.688753 12675 catalog_manager.cc:4841] TS bd71aaa953574ceea0b820af3101112d (127.25.71.67:43861): tablet e25e22ef9fd947059f0aba837e747552 (table test-table [id=b54ac0bdf40a445cbd8a212c238fc718]) successfully deleted
I20260928 16:43:41.690428 13191 ts_tablet_manager.cc:1918] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:43:41.690582 13191 log.cc:1196] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/e25e22ef9fd947059f0aba837e747552
I20260928 16:43:41.690966 13191 ts_tablet_manager.cc:1939] T e25e22ef9fd947059f0aba837e747552 P 0fbc34a19aed4418b5cf9830be6b6254: Deleting consensus metadata
W20260928 16:43:41.691409 12674 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.71.65:44171 (error 108)
I20260928 16:43:41.691890 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 12887
I20260928 16:43:41.694314 13165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.693878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:44171 (local address 127.0.0.1:34436)
0928 16:43:41.694084 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:41.694197 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.65:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 16:43:41.694589 12674 catalog_manager.cc:4584] TS e476f36ab99946808e2afc5681b0b721 (127.25.71.65:44171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e25e22ef9fd947059f0aba837e747552: Network error: Client connection negotiation failed: client connection to 127.25.71.65:44171: connect: Connection refused (error 111)
I20260928 16:43:41.699124 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 13022
I20260928 16:43:41.701303 13165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.700951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:38921 (local address 127.0.0.1:60936)
0928 16:43:41.701099 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:41.701163 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.66:38921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 16:43:41.706352 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 12658
2026-09-28T16:43:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2462 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T16:43:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:41Z Disabled control of system clock
I20260928 16:43:41.738950 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:40417
--webserver_interface=127.25.71.126
--webserver_port=0
--builtin_ntp_servers=127.25.71.84:46837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:40417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:43:41.860440 13197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:41.860741 13197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:41.860778 13197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:41.864943 13197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:41.865025 13197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:41.865039 13197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:41.865056 13197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:41.865069 13197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:41.870433 13197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:46837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:40417
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:40417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:41.871651 13197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:41.873018 13197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:41.882758 13203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:41.882783 13205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:41.882851 13197 server_base.cc:1034] running on GCE node
W20260928 16:43:41.882869 13202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:41.884119 13197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:41.885340 13197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:41.886555 13197 hybrid_clock.cc:648] HybridClock initialized: now 1790613821886537 us; error 45 us; skew 500 ppm
I20260928 16:43:41.889283 13197 webserver.cc:460] Webserver started at http://127.25.71.126:45205/ using document root <none> and password file <none>
I20260928 16:43:41.890122 13197 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:41.890210 13197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:41.890477 13197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:41.892978 13197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "497ed8ca8dd641b09fde2b5b0a127950"
format_stamp: "Formatted at 2026-09-28 16:43:41 on dist-test-slave-nqgh"
server_key: "236aa37d5ed8c5203dc554b10f51055f"
server_key_iv: "577e32d33e17d111ed0bb0ce21d9601d"
server_key_version: "encryptionkey@0"
I20260928 16:43:41.893592 13197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "497ed8ca8dd641b09fde2b5b0a127950"
format_stamp: "Formatted at 2026-09-28 16:43:41 on dist-test-slave-nqgh"
server_key: "236aa37d5ed8c5203dc554b10f51055f"
server_key_iv: "577e32d33e17d111ed0bb0ce21d9601d"
server_key_version: "encryptionkey@0"
I20260928 16:43:41.898092 13197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 16:43:41.901404 13211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:41.903203 13197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 16:43:41.903398 13197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "497ed8ca8dd641b09fde2b5b0a127950"
format_stamp: "Formatted at 2026-09-28 16:43:41 on dist-test-slave-nqgh"
server_key: "236aa37d5ed8c5203dc554b10f51055f"
server_key_iv: "577e32d33e17d111ed0bb0ce21d9601d"
server_key_version: "encryptionkey@0"
I20260928 16:43:41.903527 13197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:41.920377 13197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:41.921162 13197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:41.921363 13197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:41.929355 13197 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:40417
I20260928 16:43:41.929363 13263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:40417 every 8 connection(s)
I20260928 16:43:41.930426 13197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:41.934842 13264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:41.938310 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 13197
I20260928 16:43:41.938500 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:43:41.938930 25885 external_mini_cluster.cc:1428] Setting key 0940895774f2ef0a17ef7e9b257b2f75
I20260928 16:43:41.943265 13264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950: Bootstrap starting.
I20260928 16:43:41.946717 13264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:41.947831 13264 log.cc:824] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:41.951116 13264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950: No bootstrap required, opened a new log
I20260928 16:43:41.951133 13267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.940890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43360 (local address 127.25.71.126:40417)
0928 16:43:41.941626 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:41.941644 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:41.941686 (+ 42us) server_negotiation.cc:407] Connection header received
0928 16:43:41.942519 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:41.942539 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:41.943090 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:41.943454 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:41.944053 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.945131 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.946540 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.947453 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.947633 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:41.948713 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:41.948721 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:41.948742 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:41.949338 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:41.949419 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:41.949443 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:41.949544 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:41.949789 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 16:43:41.950192 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":142,"threads_started":1}
I20260928 16:43:41.956310 13264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497ed8ca8dd641b09fde2b5b0a127950" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 40417 } }
I20260928 16:43:41.956677 13264 raft_consensus.cc:374] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:41.956736 13264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 497ed8ca8dd641b09fde2b5b0a127950, State: Initialized, Role: FOLLOWER
I20260928 16:43:41.957320 13264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497ed8ca8dd641b09fde2b5b0a127950" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 40417 } }
I20260928 16:43:41.957481 13264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:41.957540 13264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:41.957659 13264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:41.959075 13264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497ed8ca8dd641b09fde2b5b0a127950" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 40417 } }
I20260928 16:43:41.959563 13264 leader_election.cc:302] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 497ed8ca8dd641b09fde2b5b0a127950; no voters:
I20260928 16:43:41.960001 13264 leader_election.cc:288] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:41.960237 13269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:41.960693 13269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950 [term 1 LEADER]: Becoming Leader. State: Replica: 497ed8ca8dd641b09fde2b5b0a127950, State: Running, Role: LEADER
I20260928 16:43:41.961099 13269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497ed8ca8dd641b09fde2b5b0a127950" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 40417 } }
I20260928 16:43:41.961673 13264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:41.962996 13270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "497ed8ca8dd641b09fde2b5b0a127950" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497ed8ca8dd641b09fde2b5b0a127950" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 40417 } } }
I20260928 16:43:41.963176 13270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:41.963382 13271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 497ed8ca8dd641b09fde2b5b0a127950. Latest consensus state: current_term: 1 leader_uuid: "497ed8ca8dd641b09fde2b5b0a127950" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497ed8ca8dd641b09fde2b5b0a127950" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 40417 } } }
I20260928 16:43:41.963531 13271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:41.964236 13278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:41.968453 13278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:41.976436 13278 catalog_manager.cc:1347] Generated new cluster ID: 985fc70c85d44bbcb6db7acce2463363
I20260928 16:43:41.976576 13278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:41.994990 13278 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:41.996352 13278 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:42.018873 13278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950: Generated new TSK 0
I20260928 16:43:42.020114 13278 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:43:42.036126 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.65:0
--local_ip_for_outbound_sockets=127.25.71.65
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:40417
--builtin_ntp_servers=127.25.71.84:46837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:43:42.185860 13288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:42.186180 13288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:42.186221 13288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:42.190527 13288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:42.190618 13288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:42.190706 13288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.65
I20260928 16:43:42.195858 13288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:46837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.71.65
--webserver_port=0
--tserver_master_addrs=127.25.71.126:40417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.65
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:42.197093 13288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:42.198606 13288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:42.209898 13294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:42.209820 13296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:42.210057 13293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:42.210244 13288 server_base.cc:1034] running on GCE node
I20260928 16:43:42.210974 13288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:42.212031 13288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:42.213261 13288 hybrid_clock.cc:648] HybridClock initialized: now 1790613822213235 us; error 42 us; skew 500 ppm
I20260928 16:43:42.216041 13288 webserver.cc:460] Webserver started at http://127.25.71.65:37071/ using document root <none> and password file <none>
I20260928 16:43:42.216899 13288 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:42.216985 13288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:42.217223 13288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:42.220299 13288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-0/data/instance:
uuid: "781e43b93aaa4f498e7003364289cae1"
format_stamp: "Formatted at 2026-09-28 16:43:42 on dist-test-slave-nqgh"
server_key: "fbcd1440fd8bb09e134eee3ebc0f230d"
server_key_iv: "c4b833a3007043ce57e467cb9d82462e"
server_key_version: "encryptionkey@0"
I20260928 16:43:42.221071 13288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance:
uuid: "781e43b93aaa4f498e7003364289cae1"
format_stamp: "Formatted at 2026-09-28 16:43:42 on dist-test-slave-nqgh"
server_key: "fbcd1440fd8bb09e134eee3ebc0f230d"
server_key_iv: "c4b833a3007043ce57e467cb9d82462e"
server_key_version: "encryptionkey@0"
I20260928 16:43:42.226717 13288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 16:43:42.230564 13302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:42.232563 13288 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:43:42.232811 13288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-0/wal
uuid: "781e43b93aaa4f498e7003364289cae1"
format_stamp: "Formatted at 2026-09-28 16:43:42 on dist-test-slave-nqgh"
server_key: "fbcd1440fd8bb09e134eee3ebc0f230d"
server_key_iv: "c4b833a3007043ce57e467cb9d82462e"
server_key_version: "encryptionkey@0"
I20260928 16:43:42.232975 13288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:42.267236 13288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:42.268252 13288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:42.268455 13288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:42.269212 13288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:42.270490 13288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:42.270545 13288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:42.270674 13288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:42.270714 13288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 16:43:42.283660 13288 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.65:34437
I20260928 16:43:42.283777 13415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.65:34437 every 8 connection(s)
I20260928 16:43:42.284988 13288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-0/data/info.pb
I20260928 16:43:42.286993 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 13288
I20260928 16:43:42.287281 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-0/wal/instance
I20260928 16:43:42.287734 25885 external_mini_cluster.cc:1428] Setting key d1e73e6ad7a19ab43964c41496250927
I20260928 16:43:42.290242 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.66:0
--local_ip_for_outbound_sockets=127.25.71.66
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:40417
--builtin_ntp_servers=127.25.71.84:46837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:43:42.298429 13267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.287525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.65:34765 (local address 127.25.71.126:40417)
0928 16:43:42.287664 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:42.287668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:42.289669 (+ 2001us) server_negotiation.cc:407] Connection header received
0928 16:43:42.291611 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:42.291616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:42.291699 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:42.291788 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:42.293091 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.293929 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.295771 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.296197 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.296371 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.297708 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:42.297714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:42.297717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:42.297743 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:42.297777 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:42.297791 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:42.297900 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:42.298080 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 16:43:42.298186 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:43:42.299273 13418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.288309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:40417 (local address 127.25.71.65:34765)
0928 16:43:42.289445 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:42.289516 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:42.291288 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:42.292097 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:42.292107 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:42.292229 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:42.292901 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:42.292918 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.294208 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.294214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:42.295553 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:42.295577 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.296437 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.296440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:42.296633 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.297349 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:43:42.297350 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:42.297398 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:43:42.297495 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:42.297914 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:42.297922 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:42.297927 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:43:42.298047 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 16:43:42.298318 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":251,"threads_started":1}
I20260928 16:43:42.301368 13416 heartbeater.cc:347] Connected to a master server at 127.25.71.126:40417
I20260928 16:43:42.301683 13416 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:42.302364 13416 heartbeater.cc:511] Master 127.25.71.126:40417 requested a full tablet report, sending...
I20260928 16:43:42.304240 13225 ts_manager.cc:194] Registered new tserver with Master: 781e43b93aaa4f498e7003364289cae1 (127.25.71.65:34437)
I20260928 16:43:42.305544 13225 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.71.65:34765
W20260928 16:43:42.432206 13419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:42.432623 13419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:42.432686 13419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:42.437394 13419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:42.437500 13419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:42.437597 13419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.66
I20260928 16:43:42.443694 13419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:46837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.71.66
--webserver_port=0
--tserver_master_addrs=127.25.71.126:40417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.66
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:42.445346 13419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:42.447296 13419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:42.459883 13424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:42.459878 13427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:42.459957 13425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:42.460440 13419 server_base.cc:1034] running on GCE node
I20260928 16:43:42.460985 13419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:42.461751 13419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:42.463006 13419 hybrid_clock.cc:648] HybridClock initialized: now 1790613822462969 us; error 68 us; skew 500 ppm
I20260928 16:43:42.465161 13419 webserver.cc:460] Webserver started at http://127.25.71.66:45377/ using document root <none> and password file <none>
I20260928 16:43:42.465847 13419 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:42.465926 13419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:42.466169 13419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:42.468514 13419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-1/data/instance:
uuid: "16206a92e2c54a7db4ad9958c19d1364"
format_stamp: "Formatted at 2026-09-28 16:43:42 on dist-test-slave-nqgh"
server_key: "8bf0ac4b4f1ffba793baa4a63b1c4035"
server_key_iv: "caa795a38c6a42b9fa3821dec754a27d"
server_key_version: "encryptionkey@0"
I20260928 16:43:42.469139 13419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance:
uuid: "16206a92e2c54a7db4ad9958c19d1364"
format_stamp: "Formatted at 2026-09-28 16:43:42 on dist-test-slave-nqgh"
server_key: "8bf0ac4b4f1ffba793baa4a63b1c4035"
server_key_iv: "caa795a38c6a42b9fa3821dec754a27d"
server_key_version: "encryptionkey@0"
I20260928 16:43:42.473755 13419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260928 16:43:42.477088 13433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:42.479100 13419 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:43:42.479278 13419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-1/wal
uuid: "16206a92e2c54a7db4ad9958c19d1364"
format_stamp: "Formatted at 2026-09-28 16:43:42 on dist-test-slave-nqgh"
server_key: "8bf0ac4b4f1ffba793baa4a63b1c4035"
server_key_iv: "caa795a38c6a42b9fa3821dec754a27d"
server_key_version: "encryptionkey@0"
I20260928 16:43:42.479417 13419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:42.494395 13419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:42.495196 13419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:42.495388 13419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:42.496017 13419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:42.497083 13419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:42.497134 13419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:42.497190 13419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:42.497220 13419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:42.507575 13419 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.66:40351
I20260928 16:43:42.507670 13546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.66:40351 every 8 connection(s)
I20260928 16:43:42.508756 13419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-1/data/info.pb
I20260928 16:43:42.509935 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 13419
I20260928 16:43:42.510040 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-1/wal/instance
I20260928 16:43:42.510325 25885 external_mini_cluster.cc:1428] Setting key a1da86616535d18db9908e8c11366a1f
I20260928 16:43:42.513347 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.71.67:0
--local_ip_for_outbound_sockets=127.25.71.67
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:40417
--builtin_ntp_servers=127.25.71.84:46837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:43:42.522066 13267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.511252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:33901 (local address 127.25.71.126:40417)
0928 16:43:42.511424 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:42.511430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:42.513033 (+ 1603us) server_negotiation.cc:407] Connection header received
0928 16:43:42.515247 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:42.515252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:42.515340 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:42.515481 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:42.516752 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.517378 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.519014 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.519442 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.519618 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.521360 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:42.521364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:42.521368 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:42.521391 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:42.521426 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:42.521438 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:42.521540 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:42.521767 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 16:43:42.521865 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:43:42.523402 13549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.511643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:40417 (local address 127.25.71.66:33901)
0928 16:43:42.512806 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:42.512882 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:42.514890 (+ 2008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:42.515726 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:42.515736 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:42.515858 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:42.516540 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:42.516557 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.517552 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.517555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:42.518814 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:42.518826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.519722 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.519727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:42.519933 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.520844 (+ 911us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:43:42.520846 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:42.520909 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:43:42.521009 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:42.521578 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:42.521589 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:42.521595 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:43:42.521744 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 16:43:42.522112 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":367,"threads_started":1}
I20260928 16:43:42.525055 13547 heartbeater.cc:347] Connected to a master server at 127.25.71.126:40417
I20260928 16:43:42.525434 13547 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:42.526208 13547 heartbeater.cc:511] Master 127.25.71.126:40417 requested a full tablet report, sending...
I20260928 16:43:42.527693 13225 ts_manager.cc:194] Registered new tserver with Master: 16206a92e2c54a7db4ad9958c19d1364 (127.25.71.66:40351)
I20260928 16:43:42.528507 13225 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.71.66:33901
W20260928 16:43:42.648276 13550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:42.648700 13550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:42.648765 13550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:42.653556 13550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:42.653661 13550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:42.653751 13550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.71.67
I20260928 16:43:42.659843 13550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:46837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.71.67
--webserver_port=0
--tserver_master_addrs=127.25.71.126:40417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.71.67
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:42.661473 13550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:42.663527 13550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:42.677784 13555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:42.677919 13558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:42.677855 13550 server_base.cc:1034] running on GCE node
W20260928 16:43:42.677793 13556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:42.678730 13550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:42.679570 13550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:42.680881 13550 hybrid_clock.cc:648] HybridClock initialized: now 1790613822680856 us; error 60 us; skew 500 ppm
I20260928 16:43:42.683426 13550 webserver.cc:460] Webserver started at http://127.25.71.67:40249/ using document root <none> and password file <none>
I20260928 16:43:42.684139 13550 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:42.684223 13550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:42.684425 13550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:42.686726 13550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-2/data/instance:
uuid: "a4fd6598bfdb41e9a9878a3b90578bb6"
format_stamp: "Formatted at 2026-09-28 16:43:42 on dist-test-slave-nqgh"
server_key: "cf3261640c86fa4fe5d1bff8f23f8eac"
server_key_iv: "bda824e548f178bfb554f79f60bdbf3e"
server_key_version: "encryptionkey@0"
I20260928 16:43:42.687305 13550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance:
uuid: "a4fd6598bfdb41e9a9878a3b90578bb6"
format_stamp: "Formatted at 2026-09-28 16:43:42 on dist-test-slave-nqgh"
server_key: "cf3261640c86fa4fe5d1bff8f23f8eac"
server_key_iv: "bda824e548f178bfb554f79f60bdbf3e"
server_key_version: "encryptionkey@0"
I20260928 16:43:42.691972 13550 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:43:42.695516 13564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:42.697340 13550 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:43:42.697501 13550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-2/wal
uuid: "a4fd6598bfdb41e9a9878a3b90578bb6"
format_stamp: "Formatted at 2026-09-28 16:43:42 on dist-test-slave-nqgh"
server_key: "cf3261640c86fa4fe5d1bff8f23f8eac"
server_key_iv: "bda824e548f178bfb554f79f60bdbf3e"
server_key_version: "encryptionkey@0"
I20260928 16:43:42.697660 13550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:42.714905 13550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:42.715721 13550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:42.715917 13550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:42.716575 13550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:42.717689 13550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:42.717746 13550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:42.717803 13550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:42.717832 13550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:42.729856 13550 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.67:40195
I20260928 16:43:42.729905 13677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.67:40195 every 8 connection(s)
I20260928 16:43:42.731210 13550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-2/data/info.pb
I20260928 16:43:42.732685 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 13550
I20260928 16:43:42.732815 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-2/wal/instance
I20260928 16:43:42.733127 25885 external_mini_cluster.cc:1428] Setting key e5184b4e26acd065cffb95d2d815a486
I20260928 16:43:42.743090 13267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.733865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.67:56225 (local address 127.25.71.126:40417)
0928 16:43:42.734006 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:42.734010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:42.735996 (+ 1986us) server_negotiation.cc:407] Connection header received
0928 16:43:42.737595 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:42.737599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:42.737679 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:42.737790 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:42.738932 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.739577 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.740871 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.741212 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.741319 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.742504 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:42.742507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:42.742510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:42.742529 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:42.742556 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:42.742564 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:42.742666 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:42.742856 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 16:43:42.742935 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:43:42.743959 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.734533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:40417 (local address 127.25.71.67:56225)
0928 16:43:42.735751 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:42.735831 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:42.737301 (+ 1470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:42.738035 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:42.738042 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:42.738141 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:42.738749 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:42.738766 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.739719 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.739722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:42.740689 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:42.740700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.741393 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.741396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:42.741568 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.742215 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:43:42.742217 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:42.742260 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:43:42.742339 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:42.742688 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:42.742701 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:42.742706 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:43:42.742823 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 16:43:42.743088 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":223,"threads_started":1}
I20260928 16:43:42.745946 13678 heartbeater.cc:347] Connected to a master server at 127.25.71.126:40417
I20260928 16:43:42.746284 13678 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:42.747010 13678 heartbeater.cc:511] Master 127.25.71.126:40417 requested a full tablet report, sending...
I20260928 16:43:42.748528 13225 ts_manager.cc:194] Registered new tserver with Master: a4fd6598bfdb41e9a9878a3b90578bb6 (127.25.71.67:40195)
I20260928 16:43:42.749244 13225 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.71.67:56225
I20260928 16:43:42.761329 25885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:43:42.768725 13267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.764105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.25.71.126:40417)
0928 16:43:42.764305 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:42.764311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:42.764493 (+ 182us) server_negotiation.cc:407] Connection header received
0928 16:43:42.764658 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:42.764662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:42.764718 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:42.764788 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:42.765140 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.765763 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.766830 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.767096 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.767194 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.768231 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:42.768234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:42.768236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:42.768252 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:42.768280 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:42.768288 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:42.768351 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:42.768485 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 16:43:42.768560 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 16:43:42.772753 13225 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:43:42.776609 13225 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:43:42.795046 13693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.788180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:34437 (local address 127.0.0.1:34392)
0928 16:43:42.789460 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:42.789474 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:42.789571 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:42.790597 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:42.790603 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:42.790669 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:42.790872 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:42.790881 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.792018 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.792022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:42.793372 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:42.793385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.794370 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.794373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:42.794514 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.794556 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 16:43:42.794709 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 16:43:42.794874 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1210,"mutex_wait_us":255,"spinlock_wait_cycles":6272,"thread_start_us":235,"threads_started":1}
I20260928 16:43:42.795047 13691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.788479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34392 (local address 127.25.71.65:34437)
0928 16:43:42.790050 (+ 1571us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:42.790057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:42.790089 (+ 32us) server_negotiation.cc:407] Connection header received
0928 16:43:42.790188 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:42.790193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:42.790430 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:42.790771 (+ 341us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:42.790993 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.791862 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.793594 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.794219 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.794346 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.794604 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:42.794744 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 16:43:42.794873 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1422,"thread_start_us":185,"threads_started":1}
I20260928 16:43:42.796422 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.788176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.66:40351 (local address 127.0.0.1:36948)
0928 16:43:42.789222 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:42.789304 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:42.789536 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:42.790596 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:42.790603 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:42.790663 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:42.791096 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:42.791104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.792734 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.792738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:42.794214 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:42.794228 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.795864 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.795870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:42.795987 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.796016 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 16:43:42.796093 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 16:43:42.796210 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":182,"spinlock_wait_cycles":768,"thread_start_us":96,"threads_started":1}
I20260928 16:43:42.797094 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.788432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36948 (local address 127.25.71.66:40351)
0928 16:43:42.788995 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:42.789009 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:42.789745 (+ 736us) server_negotiation.cc:407] Connection header received
0928 16:43:42.789869 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:42.789876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:42.790343 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:42.790587 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:42.791246 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.792612 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.794384 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.795397 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.795516 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.795773 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:42.796735 (+ 962us) server_negotiation.cc:299] Negotiation successful
0928 16:43:42.796898 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":193,"threads_started":1}
I20260928 16:43:42.797284 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.789095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:40195 (local address 127.0.0.1:54962)
0928 16:43:42.790101 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:42.790118 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:42.790251 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:42.790994 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:42.790997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:42.791017 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:42.791124 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:42.791129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.792720 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.792724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:42.794214 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:42.794228 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.796675 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.796679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:42.796793 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.796822 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 16:43:42.796886 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 16:43:42.797096 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":95,"thread_start_us":129,"threads_started":1}
I20260928 16:43:42.798164 13694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.789315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54962 (local address 127.25.71.67:40195)
0928 16:43:42.789938 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:42.789954 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:42.790290 (+ 336us) server_negotiation.cc:407] Connection header received
0928 16:43:42.790394 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:42.790399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:42.790612 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:42.790842 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:42.791248 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.792540 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.794385 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.796225 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.796369 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.797755 (+ 1386us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:42.797830 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 16:43:42.798025 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":299,"threads_started":1}
I20260928 16:43:42.800033 13350 tablet_service.cc:1467] Processing CreateTablet for tablet 8af697c1b62a4ade81dde367851e7f14 (DEFAULT_TABLE table=test-table [id=6e8c947a6c354e0b848ac62ac6a9207a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:42.800204 13481 tablet_service.cc:1467] Processing CreateTablet for tablet 8af697c1b62a4ade81dde367851e7f14 (DEFAULT_TABLE table=test-table [id=6e8c947a6c354e0b848ac62ac6a9207a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:42.801638 13481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af697c1b62a4ade81dde367851e7f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:42.801640 13350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af697c1b62a4ade81dde367851e7f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:42.801874 13612 tablet_service.cc:1467] Processing CreateTablet for tablet 8af697c1b62a4ade81dde367851e7f14 (DEFAULT_TABLE table=test-table [id=6e8c947a6c354e0b848ac62ac6a9207a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:42.803104 13612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af697c1b62a4ade81dde367851e7f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:42.812227 13695 tablet_bootstrap.cc:492] T 8af697c1b62a4ade81dde367851e7f14 P a4fd6598bfdb41e9a9878a3b90578bb6: Bootstrap starting.
I20260928 16:43:42.812336 13697 tablet_bootstrap.cc:492] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364: Bootstrap starting.
I20260928 16:43:42.813289 13696 tablet_bootstrap.cc:492] T 8af697c1b62a4ade81dde367851e7f14 P 781e43b93aaa4f498e7003364289cae1: Bootstrap starting.
I20260928 16:43:42.815577 13697 tablet_bootstrap.cc:654] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:42.815577 13695 tablet_bootstrap.cc:654] T 8af697c1b62a4ade81dde367851e7f14 P a4fd6598bfdb41e9a9878a3b90578bb6: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:42.815862 13696 tablet_bootstrap.cc:654] T 8af697c1b62a4ade81dde367851e7f14 P 781e43b93aaa4f498e7003364289cae1: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:42.816643 13697 log.cc:824] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:42.816643 13695 log.cc:824] T 8af697c1b62a4ade81dde367851e7f14 P a4fd6598bfdb41e9a9878a3b90578bb6: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:42.816843 13696 log.cc:824] T 8af697c1b62a4ade81dde367851e7f14 P 781e43b93aaa4f498e7003364289cae1: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:42.819458 13695 tablet_bootstrap.cc:492] T 8af697c1b62a4ade81dde367851e7f14 P a4fd6598bfdb41e9a9878a3b90578bb6: No bootstrap required, opened a new log
I20260928 16:43:42.819630 13697 tablet_bootstrap.cc:492] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364: No bootstrap required, opened a new log
I20260928 16:43:42.819934 13697 ts_tablet_manager.cc:1397] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 16:43:42.819971 13696 tablet_bootstrap.cc:492] T 8af697c1b62a4ade81dde367851e7f14 P 781e43b93aaa4f498e7003364289cae1: No bootstrap required, opened a new log
I20260928 16:43:42.820065 13695 ts_tablet_manager.cc:1397] T 8af697c1b62a4ade81dde367851e7f14 P a4fd6598bfdb41e9a9878a3b90578bb6: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 16:43:42.820498 13696 ts_tablet_manager.cc:1397] T 8af697c1b62a4ade81dde367851e7f14 P 781e43b93aaa4f498e7003364289cae1: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:43:42.823828 13695 raft_consensus.cc:348] T 8af697c1b62a4ade81dde367851e7f14 P a4fd6598bfdb41e9a9878a3b90578bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e43b93aaa4f498e7003364289cae1" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34437 } } peers { permanent_uuid: "16206a92e2c54a7db4ad9958c19d1364" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40351 } } peers { permanent_uuid: "a4fd6598bfdb41e9a9878a3b90578bb6" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 40195 } }
I20260928 16:43:42.824128 13695 raft_consensus.cc:374] T 8af697c1b62a4ade81dde367851e7f14 P a4fd6598bfdb41e9a9878a3b90578bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:42.824178 13695 raft_consensus.cc:729] T 8af697c1b62a4ade81dde367851e7f14 P a4fd6598bfdb41e9a9878a3b90578bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4fd6598bfdb41e9a9878a3b90578bb6, State: Initialized, Role: FOLLOWER
I20260928 16:43:42.824664 13695 consensus_queue.cc:260] T 8af697c1b62a4ade81dde367851e7f14 P a4fd6598bfdb41e9a9878a3b90578bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e43b93aaa4f498e7003364289cae1" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34437 } } peers { permanent_uuid: "16206a92e2c54a7db4ad9958c19d1364" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40351 } } peers { permanent_uuid: "a4fd6598bfdb41e9a9878a3b90578bb6" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 40195 } }
I20260928 16:43:42.825390 13678 heartbeater.cc:503] Master 127.25.71.126:40417 was elected leader, sending a full tablet report...
I20260928 16:43:42.825194 13697 raft_consensus.cc:348] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e43b93aaa4f498e7003364289cae1" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34437 } } peers { permanent_uuid: "16206a92e2c54a7db4ad9958c19d1364" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40351 } } peers { permanent_uuid: "a4fd6598bfdb41e9a9878a3b90578bb6" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 40195 } }
I20260928 16:43:42.825279 13696 raft_consensus.cc:348] T 8af697c1b62a4ade81dde367851e7f14 P 781e43b93aaa4f498e7003364289cae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e43b93aaa4f498e7003364289cae1" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34437 } } peers { permanent_uuid: "16206a92e2c54a7db4ad9958c19d1364" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40351 } } peers { permanent_uuid: "a4fd6598bfdb41e9a9878a3b90578bb6" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 40195 } }
I20260928 16:43:42.825587 13696 raft_consensus.cc:374] T 8af697c1b62a4ade81dde367851e7f14 P 781e43b93aaa4f498e7003364289cae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:42.825587 13697 raft_consensus.cc:374] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:42.825652 13696 raft_consensus.cc:729] T 8af697c1b62a4ade81dde367851e7f14 P 781e43b93aaa4f498e7003364289cae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 781e43b93aaa4f498e7003364289cae1, State: Initialized, Role: FOLLOWER
I20260928 16:43:42.825654 13697 raft_consensus.cc:729] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16206a92e2c54a7db4ad9958c19d1364, State: Initialized, Role: FOLLOWER
I20260928 16:43:42.826228 13697 consensus_queue.cc:260] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e43b93aaa4f498e7003364289cae1" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34437 } } peers { permanent_uuid: "16206a92e2c54a7db4ad9958c19d1364" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40351 } } peers { permanent_uuid: "a4fd6598bfdb41e9a9878a3b90578bb6" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 40195 } }
I20260928 16:43:42.826228 13696 consensus_queue.cc:260] T 8af697c1b62a4ade81dde367851e7f14 P 781e43b93aaa4f498e7003364289cae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e43b93aaa4f498e7003364289cae1" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34437 } } peers { permanent_uuid: "16206a92e2c54a7db4ad9958c19d1364" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40351 } } peers { permanent_uuid: "a4fd6598bfdb41e9a9878a3b90578bb6" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 40195 } }
I20260928 16:43:42.827487 13416 heartbeater.cc:503] Master 127.25.71.126:40417 was elected leader, sending a full tablet report...
I20260928 16:43:42.827736 13697 ts_tablet_manager.cc:1428] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260928 16:43:42.828893 13547 heartbeater.cc:503] Master 127.25.71.126:40417 was elected leader, sending a full tablet report...
I20260928 16:43:42.829655 13696 ts_tablet_manager.cc:1428] T 8af697c1b62a4ade81dde367851e7f14 P 781e43b93aaa4f498e7003364289cae1: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260928 16:43:42.830745 13695 ts_tablet_manager.cc:1428] T 8af697c1b62a4ade81dde367851e7f14 P a4fd6598bfdb41e9a9878a3b90578bb6: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260928 16:43:42.845737 13702 raft_consensus.cc:482] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:42.846161 13702 raft_consensus.cc:504] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e43b93aaa4f498e7003364289cae1" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34437 } } peers { permanent_uuid: "16206a92e2c54a7db4ad9958c19d1364" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40351 } } peers { permanent_uuid: "a4fd6598bfdb41e9a9878a3b90578bb6" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 40195 } }
I20260928 16:43:42.847793 13702 leader_election.cc:288] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 781e43b93aaa4f498e7003364289cae1 (127.25.71.65:34437), a4fd6598bfdb41e9a9878a3b90578bb6 (127.25.71.67:40195)
I20260928 16:43:42.853914 13549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.848206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:34437 (local address 127.25.71.66:38815)
0928 16:43:42.848376 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:42.848393 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:42.848505 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:42.849065 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:42.849068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:42.849089 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:42.849256 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:42.849264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.850397 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.850400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:42.852012 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:42.852026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.853162 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.853166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:42.853293 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.853396 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 16:43:42.853492 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 16:43:42.853657 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":110,"spinlock_wait_cycles":10624}
I20260928 16:43:42.854082 13694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.848712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:50803 (local address 127.25.71.67:40195)
0928 16:43:42.848883 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:42.848888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:42.849118 (+ 230us) server_negotiation.cc:407] Connection header received
0928 16:43:42.849249 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:42.849251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:42.849296 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:42.849379 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:42.849665 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.850400 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.852222 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.852970 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.853154 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.853226 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:42.853808 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 16:43:42.853946 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:43:42.854338 13691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.848461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.71.66:38815 (local address 127.25.71.65:34437)
0928 16:43:42.848708 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:42.848712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:42.848730 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:43:42.848819 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:42.848822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:42.848895 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:42.849014 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:42.849375 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.850217 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.852222 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.852967 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.853154 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.853226 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:42.854049 (+ 823us) server_negotiation.cc:299] Negotiation successful
0928 16:43:42.854169 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 16:43:42.854681 13704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.848527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.67:40195 (local address 127.25.71.66:50803)
0928 16:43:42.849003 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:42.849022 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:42.849145 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:42.849441 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:42.849443 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:42.849460 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:42.849552 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:42.849556 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.850529 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.850532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:42.852012 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:42.852026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.853380 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.853385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:42.853513 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.853598 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 16:43:42.853713 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 16:43:42.853823 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":38,"thread_start_us":166,"threads_started":1}
I20260928 16:43:42.855230 13370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af697c1b62a4ade81dde367851e7f14" candidate_uuid: "16206a92e2c54a7db4ad9958c19d1364" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "781e43b93aaa4f498e7003364289cae1" is_pre_election: true
I20260928 16:43:42.855515 13632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af697c1b62a4ade81dde367851e7f14" candidate_uuid: "16206a92e2c54a7db4ad9958c19d1364" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4fd6598bfdb41e9a9878a3b90578bb6" is_pre_election: true
I20260928 16:43:42.855700 13370 raft_consensus.cc:2454] T 8af697c1b62a4ade81dde367851e7f14 P 781e43b93aaa4f498e7003364289cae1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16206a92e2c54a7db4ad9958c19d1364 in term 0.
I20260928 16:43:42.855865 13632 raft_consensus.cc:2454] T 8af697c1b62a4ade81dde367851e7f14 P a4fd6598bfdb41e9a9878a3b90578bb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16206a92e2c54a7db4ad9958c19d1364 in term 0.
I20260928 16:43:42.856446 13437 leader_election.cc:302] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16206a92e2c54a7db4ad9958c19d1364, 781e43b93aaa4f498e7003364289cae1; no voters:
I20260928 16:43:42.856879 13702 raft_consensus.cc:2781] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:43:42.857009 13702 raft_consensus.cc:482] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:43:42.857057 13702 raft_consensus.cc:3037] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:42.858493 13702 raft_consensus.cc:504] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e43b93aaa4f498e7003364289cae1" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34437 } } peers { permanent_uuid: "16206a92e2c54a7db4ad9958c19d1364" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40351 } } peers { permanent_uuid: "a4fd6598bfdb41e9a9878a3b90578bb6" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 40195 } }
I20260928 16:43:42.858973 13702 leader_election.cc:288] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 [CANDIDATE]: Term 1 election: Requested vote from peers 781e43b93aaa4f498e7003364289cae1 (127.25.71.65:34437), a4fd6598bfdb41e9a9878a3b90578bb6 (127.25.71.67:40195)
I20260928 16:43:42.859673 13632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af697c1b62a4ade81dde367851e7f14" candidate_uuid: "16206a92e2c54a7db4ad9958c19d1364" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4fd6598bfdb41e9a9878a3b90578bb6"
I20260928 16:43:42.859678 13370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af697c1b62a4ade81dde367851e7f14" candidate_uuid: "16206a92e2c54a7db4ad9958c19d1364" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "781e43b93aaa4f498e7003364289cae1"
I20260928 16:43:42.859846 13632 raft_consensus.cc:3037] T 8af697c1b62a4ade81dde367851e7f14 P a4fd6598bfdb41e9a9878a3b90578bb6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:42.859864 13370 raft_consensus.cc:3037] T 8af697c1b62a4ade81dde367851e7f14 P 781e43b93aaa4f498e7003364289cae1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:42.861155 13632 raft_consensus.cc:2454] T 8af697c1b62a4ade81dde367851e7f14 P a4fd6598bfdb41e9a9878a3b90578bb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16206a92e2c54a7db4ad9958c19d1364 in term 1.
I20260928 16:43:42.861256 13370 raft_consensus.cc:2454] T 8af697c1b62a4ade81dde367851e7f14 P 781e43b93aaa4f498e7003364289cae1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16206a92e2c54a7db4ad9958c19d1364 in term 1.
I20260928 16:43:42.861724 13437 leader_election.cc:302] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16206a92e2c54a7db4ad9958c19d1364, a4fd6598bfdb41e9a9878a3b90578bb6; no voters:
I20260928 16:43:42.861986 13702 raft_consensus.cc:2781] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:42.862327 13702 raft_consensus.cc:686] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 [term 1 LEADER]: Becoming Leader. State: Replica: 16206a92e2c54a7db4ad9958c19d1364, State: Running, Role: LEADER
I20260928 16:43:42.862836 13702 consensus_queue.cc:237] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e43b93aaa4f498e7003364289cae1" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34437 } } peers { permanent_uuid: "16206a92e2c54a7db4ad9958c19d1364" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40351 } } peers { permanent_uuid: "a4fd6598bfdb41e9a9878a3b90578bb6" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 40195 } }
I20260928 16:43:42.866724 13226 catalog_manager.cc:5495] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16206a92e2c54a7db4ad9958c19d1364 (127.25.71.66). New cstate: current_term: 1 leader_uuid: "16206a92e2c54a7db4ad9958c19d1364" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e43b93aaa4f498e7003364289cae1" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16206a92e2c54a7db4ad9958c19d1364" member_type: VOTER last_known_addr { host: "127.25.71.66" port: 40351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4fd6598bfdb41e9a9878a3b90578bb6" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 40195 } health_report { overall_health: UNKNOWN } } }
I20260928 16:43:42.881475 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.877398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36954 (local address 127.25.71.66:40351)
0928 16:43:42.877567 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:42.877572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:42.877590 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:43:42.877691 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:42.877693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:42.877755 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:42.877862 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:42.878152 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.878826 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.879745 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.880046 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.880143 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.880591 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:42.881082 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:42.881194 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:42.881268 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 16:43:42.881327 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 16:43:42.889525 13632 raft_consensus.cc:1261] T 8af697c1b62a4ade81dde367851e7f14 P a4fd6598bfdb41e9a9878a3b90578bb6 [term 1 FOLLOWER]: Refusing update from remote peer 16206a92e2c54a7db4ad9958c19d1364: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:42.889634 13370 raft_consensus.cc:1261] T 8af697c1b62a4ade81dde367851e7f14 P 781e43b93aaa4f498e7003364289cae1 [term 1 FOLLOWER]: Refusing update from remote peer 16206a92e2c54a7db4ad9958c19d1364: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:42.890501 13702 consensus_queue.cc:1035] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4fd6598bfdb41e9a9878a3b90578bb6" member_type: VOTER last_known_addr { host: "127.25.71.67" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:42.890803 13705 consensus_queue.cc:1035] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 [LEADER]: Connected to new peer: Peer: permanent_uuid: "781e43b93aaa4f498e7003364289cae1" member_type: VOTER last_known_addr { host: "127.25.71.65" port: 34437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:42.905579 13709 mvcc.cc:205] Tried to move back new op lower bound from 7334354218544476160 to 7334354218449506304. Current Snapshot: MvccSnapshot[applied={T|T < 7334354218544476160}]
I20260928 16:43:42.905841 13710 mvcc.cc:205] Tried to move back new op lower bound from 7334354218544476160 to 7334354218449506304. Current Snapshot: MvccSnapshot[applied={T|T < 7334354218544476160}]
I20260928 16:43:42.912474 13226 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:42.912736 13226 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:42.916169 13226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 497ed8ca8dd641b09fde2b5b0a127950: Sending DeleteTablet for 3 replicas of tablet 8af697c1b62a4ade81dde367851e7f14
I20260928 16:43:42.917456 13612 tablet_service.cc:1514] Processing DeleteTablet for tablet 8af697c1b62a4ade81dde367851e7f14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:42 UTC) from {username='slave'} at 127.0.0.1:54962
I20260928 16:43:42.917459 13350 tablet_service.cc:1514] Processing DeleteTablet for tablet 8af697c1b62a4ade81dde367851e7f14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:42 UTC) from {username='slave'} at 127.0.0.1:34392
I20260928 16:43:42.917958 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 13288
I20260928 16:43:42.918426 13481 tablet_service.cc:1514] Processing DeleteTablet for tablet 8af697c1b62a4ade81dde367851e7f14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:42 UTC) from {username='slave'} at 127.0.0.1:36948
I20260928 16:43:42.919059 13720 tablet_replica.cc:330] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364: stopping tablet replica
I20260928 16:43:42.919062 13718 tablet_replica.cc:330] T 8af697c1b62a4ade81dde367851e7f14 P a4fd6598bfdb41e9a9878a3b90578bb6: stopping tablet replica
I20260928 16:43:42.919526 13720 raft_consensus.cc:2229] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:43:42.919526 13718 raft_consensus.cc:2229] T 8af697c1b62a4ade81dde367851e7f14 P a4fd6598bfdb41e9a9878a3b90578bb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:42.919986 13718 raft_consensus.cc:2258] T 8af697c1b62a4ade81dde367851e7f14 P a4fd6598bfdb41e9a9878a3b90578bb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:42.920200 13720 raft_consensus.cc:2258] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:42.922842 13720 ts_tablet_manager.cc:1905] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:42.926405 13720 ts_tablet_manager.cc:1918] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:43:42.926560 13720 log.cc:1196] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-1/wal/wals/8af697c1b62a4ade81dde367851e7f14
I20260928 16:43:42.927302 13720 ts_tablet_manager.cc:1939] T 8af697c1b62a4ade81dde367851e7f14 P 16206a92e2c54a7db4ad9958c19d1364: Deleting consensus metadata
I20260928 16:43:42.928424 13718 ts_tablet_manager.cc:1905] T 8af697c1b62a4ade81dde367851e7f14 P a4fd6598bfdb41e9a9878a3b90578bb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:42.929700 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 13419
I20260928 16:43:42.929735 13213 catalog_manager.cc:4841] TS 16206a92e2c54a7db4ad9958c19d1364 (127.25.71.66:40351): tablet 8af697c1b62a4ade81dde367851e7f14 (table test-table [id=6e8c947a6c354e0b848ac62ac6a9207a]) successfully deleted
W20260928 16:43:42.930241 13215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.71.65:34437 (error 108)
I20260928 16:43:42.932991 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.932575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.65:34437 (local address 127.0.0.1:34402)
0928 16:43:42.932751 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:42.932879 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.65:34437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 16:43:42.933259 13215 catalog_manager.cc:4584] TS 781e43b93aaa4f498e7003364289cae1 (127.25.71.65:34437): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8af697c1b62a4ade81dde367851e7f14: Network error: Client connection negotiation failed: client connection to 127.25.71.65:34437: connect: Connection refused (error 111)
I20260928 16:43:42.934190 13718 ts_tablet_manager.cc:1918] T 8af697c1b62a4ade81dde367851e7f14 P a4fd6598bfdb41e9a9878a3b90578bb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:43:42.934346 13718 log.cc:1196] T 8af697c1b62a4ade81dde367851e7f14 P a4fd6598bfdb41e9a9878a3b90578bb6: Deleting WAL directory at /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613545667819-25885-0/minicluster-data/ts-2/wal/wals/8af697c1b62a4ade81dde367851e7f14
I20260928 16:43:42.934835 13718 ts_tablet_manager.cc:1939] T 8af697c1b62a4ade81dde367851e7f14 P a4fd6598bfdb41e9a9878a3b90578bb6: Deleting consensus metadata
I20260928 16:43:42.936431 13215 catalog_manager.cc:4841] TS a4fd6598bfdb41e9a9878a3b90578bb6 (127.25.71.67:40195): tablet 8af697c1b62a4ade81dde367851e7f14 (table test-table [id=6e8c947a6c354e0b848ac62ac6a9207a]) successfully deleted
I20260928 16:43:42.938799 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 13550
I20260928 16:43:42.946107 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 13197
2026-09-28T16:43:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1239 ms)
[----------] 6 tests from EncryptionPolicyTest (11547 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:43 dist-test-slave-nqgh krb5kdc[13724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:43 dist-test-slave-nqgh krb5kdc[13724](info): set up 2 sockets
Sep 28 16:43:43 dist-test-slave-nqgh krb5kdc[13724](info): commencing operation
krb5kdc: starting...
W20260928 16:43:43.928494 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:43:44 dist-test-slave-nqgh krb5kdc[13724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:43:44.071662 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40881
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:44.210117 13737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:44.210518 13737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:44.210568 13737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:44.215112 13737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:44.215225 13737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:44.215246 13737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:44.215263 13737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:44.215279 13737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:44.221745 13737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40881
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:44.223460 13737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:44.224833 13737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:44.235715 13743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:44.235715 13742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:44.235704 13745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:44.235919 13737 server_base.cc:1034] running on GCE node
I20260928 16:43:44.236970 13737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:44.238327 13737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:44.239557 13737 hybrid_clock.cc:648] HybridClock initialized: now 1790613824239531 us; error 48 us; skew 500 ppm
Sep 28 16:43:44 dist-test-slave-nqgh krb5kdc[13724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:44.243304 13737 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:43:44.244644 13737 webserver.cc:460] Webserver started at http://127.0.0.1:38987/ using document root <none> and password file <none>
I20260928 16:43:44.245404 13737 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:44.245486 13737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:44.245744 13737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:44.248502 13737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "1c48971a226b404983d73c56a6489b31"
format_stamp: "Formatted at 2026-09-28 16:43:44 on dist-test-slave-nqgh"
server_key: "7df11d1e943d1a027b6d1941250241a1"
server_key_iv: "0e5444c8b3ffba0292e1a8d25129463c"
server_key_version: "encryptionkey@0"
I20260928 16:43:44.249243 13737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "1c48971a226b404983d73c56a6489b31"
format_stamp: "Formatted at 2026-09-28 16:43:44 on dist-test-slave-nqgh"
server_key: "7df11d1e943d1a027b6d1941250241a1"
server_key_iv: "0e5444c8b3ffba0292e1a8d25129463c"
server_key_version: "encryptionkey@0"
I20260928 16:43:44.254333 13737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:43:44.257819 13752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:44.259630 13737 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:43:44.259819 13737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "1c48971a226b404983d73c56a6489b31"
format_stamp: "Formatted at 2026-09-28 16:43:44 on dist-test-slave-nqgh"
server_key: "7df11d1e943d1a027b6d1941250241a1"
server_key_iv: "0e5444c8b3ffba0292e1a8d25129463c"
server_key_version: "encryptionkey@0"
I20260928 16:43:44.259966 13737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:44.280335 13737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:44.284799 13737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:44.285053 13737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:44.294396 13737 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40881
I20260928 16:43:44.294595 13804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40881 every 8 connection(s)
I20260928 16:43:44.295786 13737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:44.299307 13805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:44.300843 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 13737
I20260928 16:43:44.301004 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:43:44.301342 25885 external_mini_cluster.cc:1428] Setting key 57db3734be1730285147336b0f286b8b
I20260928 16:43:44.307387 13805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31: Bootstrap starting.
I20260928 16:43:44.310241 13805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:44.311223 13805 log.cc:824] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:43:44 dist-test-slave-nqgh krb5kdc[13724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 16:43:44.314218 13805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31: No bootstrap required, opened a new log
I20260928 16:43:44.319227 13805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c48971a226b404983d73c56a6489b31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40881 } }
I20260928 16:43:44.319541 13805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:44.319604 13805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c48971a226b404983d73c56a6489b31, State: Initialized, Role: FOLLOWER
I20260928 16:43:44.320013 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:44.303359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48915 (local address 127.0.0.1:40881)
0928 16:43:44.303957 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:44.303977 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:44.304028 (+ 51us) server_negotiation.cc:407] Connection header received
0928 16:43:44.304956 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:44.304980 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:44.305462 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:44.305869 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:44.307054 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:44.308315 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:44.309655 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:44.310313 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:44.310519 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:44.313581 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:44.313588 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:44.313611 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:44.313650 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:44.315681 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:44.316442 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:44.316449 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:44.316456 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:44.316603 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:44.317085 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:44.317091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:44.317093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:44.317775 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:44.318041 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:44.318576 (+ 535us) server_negotiation.cc:299] Negotiation successful
0928 16:43:44.318909 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":167,"threads_started":1}
I20260928 16:43:44.320127 13805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c48971a226b404983d73c56a6489b31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40881 } }
I20260928 16:43:44.320286 13805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:44.320345 13805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:44.320428 13805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:44.321444 13805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c48971a226b404983d73c56a6489b31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40881 } }
I20260928 16:43:44.321849 13805 leader_election.cc:302] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c48971a226b404983d73c56a6489b31; no voters:
I20260928 16:43:44.322232 13805 leader_election.cc:288] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:44.322445 13810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:44.323078 13810 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31 [term 1 LEADER]: Becoming Leader. State: Replica: 1c48971a226b404983d73c56a6489b31, State: Running, Role: LEADER
I20260928 16:43:44.323642 13805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:44.323715 13810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c48971a226b404983d73c56a6489b31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40881 } }
I20260928 16:43:44.324380 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:43:44.325898 13812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c48971a226b404983d73c56a6489b31. Latest consensus state: current_term: 1 leader_uuid: "1c48971a226b404983d73c56a6489b31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c48971a226b404983d73c56a6489b31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40881 } } }
I20260928 16:43:44.326102 13812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:44.326267 13811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c48971a226b404983d73c56a6489b31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c48971a226b404983d73c56a6489b31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40881 } } }
I20260928 16:43:44.326692 13811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31 [sys.catalog]: This master's current role is: LEADER
W20260928 16:43:44.329898 13826 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:43:44.330024 13826 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:43:44.330135 13829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:44.331390 13829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:44.338056 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:44.327295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37827 (local address 127.0.0.1:40881)
0928 16:43:44.327435 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:44.327440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:44.328150 (+ 710us) server_negotiation.cc:407] Connection header received
0928 16:43:44.328377 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:44.328381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:44.328436 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:44.328529 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:44.329360 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:44.329943 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:44.331103 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:44.331526 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:44.331682 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:44.333692 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:44.333697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:44.333700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:44.333733 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:44.335657 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:44.336343 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:44.336347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:44.336350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:44.336453 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:44.336985 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:44.336990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:44.336992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:44.337292 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:44.337465 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:44.337755 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 16:43:44.337871 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:43:44.339717 13829 catalog_manager.cc:1347] Generated new cluster ID: 66c53f082bd747bba97e4e31d1794baa
I20260928 16:43:44.339834 13829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:44.352820 13829 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:44.354271 13829 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:44.366664 13829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c48971a226b404983d73c56a6489b31: Generated new TSK 0
I20260928 16:43:44.367631 13829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:43:44.379246 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 13737
W20260928 16:43:44.386323 13722 connection.cc:535] client connection to 127.0.0.1:40881 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40881): Connection reset by peer (error 104)
2026-09-28T16:43:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:44 dist-test-slave-nqgh krb5kdc[13836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:44 dist-test-slave-nqgh krb5kdc[13836](info): set up 2 sockets
Sep 28 16:43:44 dist-test-slave-nqgh krb5kdc[13836](info): commencing operation
krb5kdc: starting...
W20260928 16:43:45.360769 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:43:45 dist-test-slave-nqgh krb5kdc[13836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:43:45.503757 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33009
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:45.640539 13849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:45.640954 13849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:45.641027 13849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:45.645830 13849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:45.645953 13849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:45.645984 13849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:45.646008 13849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:45.646028 13849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:45.652952 13849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33009
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:45.654565 13849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:45.656049 13849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:45.667114 13857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:45.667131 13854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:45.667131 13855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:45.667438 13849 server_base.cc:1034] running on GCE node
I20260928 16:43:45.668332 13849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:45.669916 13849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:45.671207 13849 hybrid_clock.cc:648] HybridClock initialized: now 1790613825671157 us; error 75 us; skew 500 ppm
Sep 28 16:43:45 dist-test-slave-nqgh krb5kdc[13836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:45.675127 13849 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:43:45.676611 13849 webserver.cc:460] Webserver started at http://127.0.0.1:39353/ using document root <none> and password file <none>
I20260928 16:43:45.677372 13849 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:45.677460 13849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:45.677731 13849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:45.680706 13849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "3393d0e2aa8f453bb54fc57dedfdb76f"
format_stamp: "Formatted at 2026-09-28 16:43:45 on dist-test-slave-nqgh"
server_key: "eeecc515b362d76850fea2527cb10040"
server_key_iv: "e2e578e5f4ab16d63ac108f46e3fca2f"
server_key_version: "encryptionkey@0"
I20260928 16:43:45.681506 13849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "3393d0e2aa8f453bb54fc57dedfdb76f"
format_stamp: "Formatted at 2026-09-28 16:43:45 on dist-test-slave-nqgh"
server_key: "eeecc515b362d76850fea2527cb10040"
server_key_iv: "e2e578e5f4ab16d63ac108f46e3fca2f"
server_key_version: "encryptionkey@0"
I20260928 16:43:45.686880 13849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 16:43:45.690613 13864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:45.692622 13849 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:43:45.692823 13849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "3393d0e2aa8f453bb54fc57dedfdb76f"
format_stamp: "Formatted at 2026-09-28 16:43:45 on dist-test-slave-nqgh"
server_key: "eeecc515b362d76850fea2527cb10040"
server_key_iv: "e2e578e5f4ab16d63ac108f46e3fca2f"
server_key_version: "encryptionkey@0"
I20260928 16:43:45.692986 13849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:45.709477 13849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:45.713987 13849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:45.714236 13849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:45.723531 13849 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33009
I20260928 16:43:45.723618 13916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33009 every 8 connection(s)
I20260928 16:43:45.724797 13849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:45.728116 13917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:45.732707 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 13849
I20260928 16:43:45.732903 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:43:45.733242 25885 external_mini_cluster.cc:1428] Setting key c4c6ef3f9948fd427ad48878569b2a6a
I20260928 16:43:45.736090 13917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f: Bootstrap starting.
I20260928 16:43:45.739768 13917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:45.740864 13917 log.cc:824] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:45.744091 13917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f: No bootstrap required, opened a new log
I20260928 16:43:45.746345 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:45.735455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44805 (local address 127.0.0.1:33009)
0928 16:43:45.736044 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:45.736065 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:45.736115 (+ 50us) server_negotiation.cc:407] Connection header received
0928 16:43:45.737043 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:45.737067 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:45.737562 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:45.737980 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:45.738578 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:45.739956 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:45.741350 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:45.742008 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:45.742233 (+ 225us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:45.743692 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:45.743699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:45.743719 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:45.744315 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:45.744397 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:45.744427 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:45.744566 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:45.744808 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 16:43:45.745003 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":152,"threads_started":1}
I20260928 16:43:45.747787 13917 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3393d0e2aa8f453bb54fc57dedfdb76f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33009 } }
I20260928 16:43:45.748083 13917 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:45.748124 13917 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3393d0e2aa8f453bb54fc57dedfdb76f, State: Initialized, Role: FOLLOWER
I20260928 16:43:45.748595 13917 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3393d0e2aa8f453bb54fc57dedfdb76f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33009 } }
I20260928 16:43:45.748739 13917 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:45.748891 13917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:45.749008 13917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:45.750145 13917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3393d0e2aa8f453bb54fc57dedfdb76f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33009 } }
I20260928 16:43:45.750299 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:43:45.750543 13917 leader_election.cc:302] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3393d0e2aa8f453bb54fc57dedfdb76f; no voters:
I20260928 16:43:45.750924 13917 leader_election.cc:288] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:45.751034 13922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:45.751242 13922 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f [term 1 LEADER]: Becoming Leader. State: Replica: 3393d0e2aa8f453bb54fc57dedfdb76f, State: Running, Role: LEADER
I20260928 16:43:45.751668 13922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3393d0e2aa8f453bb54fc57dedfdb76f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33009 } }
I20260928 16:43:45.752379 13917 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:45.753001 13922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3393d0e2aa8f453bb54fc57dedfdb76f. Latest consensus state: current_term: 1 leader_uuid: "3393d0e2aa8f453bb54fc57dedfdb76f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3393d0e2aa8f453bb54fc57dedfdb76f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33009 } } }
I20260928 16:43:45.753124 13922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:45.754237 13923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3393d0e2aa8f453bb54fc57dedfdb76f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3393d0e2aa8f453bb54fc57dedfdb76f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33009 } } }
I20260928 16:43:45.754396 13923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f [sys.catalog]: This master's current role is: LEADER
W20260928 16:43:45.758118 13942 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:43:45.758263 13942 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:43:45.758345 13932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:45.759258 13932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:45.759271 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:45.753450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53603 (local address 127.0.0.1:33009)
0928 16:43:45.753584 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:45.753590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:45.754871 (+ 1281us) server_negotiation.cc:407] Connection header received
0928 16:43:45.755132 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:45.755135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:45.755201 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:45.755320 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:45.755798 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:45.756515 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:45.757416 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:45.757672 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:45.757769 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:45.758750 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:45.758754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:45.758756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:45.758773 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:45.758804 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:45.758816 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:45.758892 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:45.759056 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 16:43:45.759105 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:43:45.767001 13932 catalog_manager.cc:1347] Generated new cluster ID: 3b6ad6fd604241f5bae3363e6790986b
I20260928 16:43:45.767161 13932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:45.777210 13932 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:45.778618 13932 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:45.795432 13932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3393d0e2aa8f453bb54fc57dedfdb76f: Generated new TSK 0
I20260928 16:43:45.796761 13932 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:43:45.811966 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 13849
W20260928 16:43:45.817853 13834 connection.cc:535] client connection to 127.0.0.1:33009 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33009): Connection reset by peer (error 104)
2026-09-28T16:43:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:45 dist-test-slave-nqgh krb5kdc[13948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:45 dist-test-slave-nqgh krb5kdc[13948](info): set up 2 sockets
Sep 28 16:43:45 dist-test-slave-nqgh krb5kdc[13948](info): commencing operation
krb5kdc: starting...
W20260928 16:43:46.788920 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:43:46 dist-test-slave-nqgh krb5kdc[13948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:43:46.931145 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:47.063511 13961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:47.063930 13961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:47.063999 13961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:47.068857 13961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:47.068974 13961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:47.069008 13961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:47.069038 13961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:47.069072 13961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:47.075577 13961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36845
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:47.077239 13961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:47.078742 13961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:47.090123 13966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:47.090103 13969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:47.090153 13967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:47.090389 13961 server_base.cc:1034] running on GCE node
I20260928 16:43:47.091662 13961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:47.093331 13961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:47.094550 13961 hybrid_clock.cc:648] HybridClock initialized: now 1790613827094512 us; error 59 us; skew 500 ppm
Sep 28 16:43:47 dist-test-slave-nqgh krb5kdc[13948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:47.105163 13961 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:43:47.106731 13961 webserver.cc:460] Webserver started at http://127.0.0.1:39443/ using document root <none> and password file <none>
I20260928 16:43:47.107506 13961 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:47.107604 13961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:47.107887 13961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:47.110859 13961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "80deedaf37fe4db79e871b61f0ffae1c"
format_stamp: "Formatted at 2026-09-28 16:43:47 on dist-test-slave-nqgh"
server_key: "4bd7597e23c17aa96e6a7d394c2ce0b6"
server_key_iv: "44d2d02ff866b4b513bd40d002d35fd7"
server_key_version: "encryptionkey@0"
I20260928 16:43:47.111660 13961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "80deedaf37fe4db79e871b61f0ffae1c"
format_stamp: "Formatted at 2026-09-28 16:43:47 on dist-test-slave-nqgh"
server_key: "4bd7597e23c17aa96e6a7d394c2ce0b6"
server_key_iv: "44d2d02ff866b4b513bd40d002d35fd7"
server_key_version: "encryptionkey@0"
I20260928 16:43:47.117152 13961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:43:47.121272 13976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:47.123785 13961 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:43:47.124043 13961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "80deedaf37fe4db79e871b61f0ffae1c"
format_stamp: "Formatted at 2026-09-28 16:43:47 on dist-test-slave-nqgh"
server_key: "4bd7597e23c17aa96e6a7d394c2ce0b6"
server_key_iv: "44d2d02ff866b4b513bd40d002d35fd7"
server_key_version: "encryptionkey@0"
I20260928 16:43:47.124269 13961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:47.130550 13961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:47.134799 13961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:47.135197 13961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:47.145116 13961 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36845
I20260928 16:43:47.145208 14028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36845 every 8 connection(s)
I20260928 16:43:47.146522 13961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:47.150514 14029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:47.150979 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 13961
I20260928 16:43:47.151136 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:43:47.151585 25885 external_mini_cluster.cc:1428] Setting key 61fd735409eb5083444057136606ca9c
I20260928 16:43:47.158767 14029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c: Bootstrap starting.
I20260928 16:43:47.159606 14032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:47.153663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41573 (local address 127.0.0.1:36845)
0928 16:43:47.154355 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:47.154395 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:47.154590 (+ 195us) server_negotiation.cc:407] Connection header received
0928 16:43:47.155665 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:47.155691 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:47.156100 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:47.156536 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:47.157796 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:47.157803 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:47.157819 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:47.157833 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:47.157916 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:47.157949 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:47.158149 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:47.158294 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 16:43:47.158523 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":193,"threads_started":1}
I20260928 16:43:47.162019 14029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:47.163312 14029 log.cc:824] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:47.164395 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:43:47.166538 14029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c: No bootstrap required, opened a new log
I20260928 16:43:47.168629 14032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:47.166708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54877 (local address 127.0.0.1:36845)
0928 16:43:47.166863 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:47.166868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:47.166973 (+ 105us) server_negotiation.cc:407] Connection header received
0928 16:43:47.167120 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:47.167123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:47.167175 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:47.167313 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:47.168149 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:47.168153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:47.168156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:47.168171 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:47.168212 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:47.168237 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:47.168344 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:47.168405 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 16:43:47.168460 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 16:43:47.171608 14029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80deedaf37fe4db79e871b61f0ffae1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } }
I20260928 16:43:47.171950 14029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:47.171990 14029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80deedaf37fe4db79e871b61f0ffae1c, State: Initialized, Role: FOLLOWER
I20260928 16:43:47.172487 14029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80deedaf37fe4db79e871b61f0ffae1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } }
I20260928 16:43:47.172624 14029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:47.172684 14029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:47.172781 14029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:47.173987 14029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80deedaf37fe4db79e871b61f0ffae1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } }
I20260928 16:43:47.174427 14029 leader_election.cc:302] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80deedaf37fe4db79e871b61f0ffae1c; no voters:
I20260928 16:43:47.174913 14029 leader_election.cc:288] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:47.175165 14039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:47.175509 14039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c [term 1 LEADER]: Becoming Leader. State: Replica: 80deedaf37fe4db79e871b61f0ffae1c, State: Running, Role: LEADER
I20260928 16:43:47.175940 14039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80deedaf37fe4db79e871b61f0ffae1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } }
I20260928 16:43:47.176541 14029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:47.177889 14041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80deedaf37fe4db79e871b61f0ffae1c. Latest consensus state: current_term: 1 leader_uuid: "80deedaf37fe4db79e871b61f0ffae1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80deedaf37fe4db79e871b61f0ffae1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } } }
I20260928 16:43:47.178082 14041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:47.178496 14040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80deedaf37fe4db79e871b61f0ffae1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80deedaf37fe4db79e871b61f0ffae1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } } }
I20260928 16:43:47.178694 14040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:47.179031 14048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:47.182611 14048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:47.189464 14048 catalog_manager.cc:1347] Generated new cluster ID: 0cb5e3d3ab994dd9aa72bd8edc52c32d
I20260928 16:43:47.189579 14048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:47.227032 14048 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:47.227443 14048 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:47.235813 14048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80deedaf37fe4db79e871b61f0ffae1c: Generated new TSK 0
I20260928 16:43:47.236794 14048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:43:47.252501 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 13961
2026-09-28T16:43:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:47 dist-test-slave-nqgh krb5kdc[14060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:47 dist-test-slave-nqgh krb5kdc[14060](info): set up 2 sockets
Sep 28 16:43:47 dist-test-slave-nqgh krb5kdc[14060](info): commencing operation
krb5kdc: starting...
W20260928 16:43:48.228991 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:43:48 dist-test-slave-nqgh krb5kdc[14060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:43:48.374713 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39403
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:48.499671 14073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:48.500001 14073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:48.500063 14073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:48.504951 14073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:48.505065 14073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:48.505095 14073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:48.505117 14073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:48.505138 14073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:48.511629 14073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:48.512900 14073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:48.514017 14073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:48.524600 14078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:48.524729 14073 server_base.cc:1034] running on GCE node
W20260928 16:43:48.524718 14079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:48.525069 14081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:48.525776 14073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:48.527146 14073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:48.528343 14073 hybrid_clock.cc:648] HybridClock initialized: now 1790613828528294 us; error 74 us; skew 500 ppm
Sep 28 16:43:48 dist-test-slave-nqgh krb5kdc[14060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:48.531653 14073 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:43:48.533020 14073 webserver.cc:460] Webserver started at http://127.0.0.1:37879/ using document root <none> and password file <none>
I20260928 16:43:48.533780 14073 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:48.533861 14073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:48.534101 14073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:48.536803 14073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "d6e61c117fd94d5ea581276ebfbae777"
format_stamp: "Formatted at 2026-09-28 16:43:48 on dist-test-slave-nqgh"
server_key: "c911dc0d9cb5122767b62253dac05205"
server_key_iv: "ada15c5b29d256df713dc1f3296dd371"
server_key_version: "encryptionkey@0"
I20260928 16:43:48.537465 14073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "d6e61c117fd94d5ea581276ebfbae777"
format_stamp: "Formatted at 2026-09-28 16:43:48 on dist-test-slave-nqgh"
server_key: "c911dc0d9cb5122767b62253dac05205"
server_key_iv: "ada15c5b29d256df713dc1f3296dd371"
server_key_version: "encryptionkey@0"
I20260928 16:43:48.542603 14073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 16:43:48.546274 14088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:48.548238 14073 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260928 16:43:48.548429 14073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "d6e61c117fd94d5ea581276ebfbae777"
format_stamp: "Formatted at 2026-09-28 16:43:48 on dist-test-slave-nqgh"
server_key: "c911dc0d9cb5122767b62253dac05205"
server_key_iv: "ada15c5b29d256df713dc1f3296dd371"
server_key_version: "encryptionkey@0"
I20260928 16:43:48.548576 14073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:48.560112 14073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:48.564054 14073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:48.564343 14073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:48.573060 14073 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39403
I20260928 16:43:48.573129 14140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39403 every 8 connection(s)
I20260928 16:43:48.574458 14073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:48.578027 14141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:48.583859 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 14073
I20260928 16:43:48.584029 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:43:48.584348 25885 external_mini_cluster.cc:1428] Setting key e33bf627b69f380d4d9c0879f0ea782f
I20260928 16:43:48.584789 14141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777: Bootstrap starting.
I20260928 16:43:48.587739 14141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:48.588587 14141 log.cc:824] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:48.590803 14141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777: No bootstrap required, opened a new log
I20260928 16:43:48.594682 14141 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e61c117fd94d5ea581276ebfbae777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39403 } }
I20260928 16:43:48.594977 14141 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:48.595046 14141 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e61c117fd94d5ea581276ebfbae777, State: Initialized, Role: FOLLOWER
Sep 28 16:43:48 dist-test-slave-nqgh krb5kdc[14060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 16:43:48.595510 14141 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e61c117fd94d5ea581276ebfbae777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39403 } }
I20260928 16:43:48.595674 14141 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:48.595743 14141 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:48.595856 14141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:48.596971 14141 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e61c117fd94d5ea581276ebfbae777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39403 } }
I20260928 16:43:48.597343 14141 leader_election.cc:302] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6e61c117fd94d5ea581276ebfbae777; no voters:
I20260928 16:43:48.597748 14141 leader_election.cc:288] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:48.597851 14146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:48.598035 14146 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777 [term 1 LEADER]: Becoming Leader. State: Replica: d6e61c117fd94d5ea581276ebfbae777, State: Running, Role: LEADER
I20260928 16:43:48.598374 14146 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e61c117fd94d5ea581276ebfbae777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39403 } }
I20260928 16:43:48.599258 14141 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:48.599714 14146 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6e61c117fd94d5ea581276ebfbae777. Latest consensus state: current_term: 1 leader_uuid: "d6e61c117fd94d5ea581276ebfbae777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e61c117fd94d5ea581276ebfbae777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39403 } } }
I20260928 16:43:48.599843 14146 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:48.600540 14147 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6e61c117fd94d5ea581276ebfbae777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e61c117fd94d5ea581276ebfbae777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39403 } } }
I20260928 16:43:48.600658 14147 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:48.601235 14150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:48.603768 14144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:48.586141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35221 (local address 127.0.0.1:39403)
0928 16:43:48.586860 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:48.586886 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:48.586950 (+ 64us) server_negotiation.cc:407] Connection header received
0928 16:43:48.588046 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:48.588081 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:48.588620 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:48.589087 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:48.590051 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:48.591311 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:48.592275 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:48.592866 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:48.593041 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:48.596225 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:48.596232 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:48.596255 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:48.596297 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:48.598857 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:48.599507 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:48.599514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:48.599521 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:48.599715 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:48.600053 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:48.600057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:48.600059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:48.600756 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:48.601013 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:48.602199 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0928 16:43:48.602606 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":198,"threads_started":1}
I20260928 16:43:48.605981 14150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:48.607530 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:43:48.614157 14150 catalog_manager.cc:1347] Generated new cluster ID: 8c00ddc90539406bb26c5527ea4a8509
I20260928 16:43:48.614279 14150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:48.620085 14144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:48.610362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37569 (local address 127.0.0.1:39403)
0928 16:43:48.610509 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:48.610514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:48.610846 (+ 332us) server_negotiation.cc:407] Connection header received
0928 16:43:48.611128 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:48.611132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:48.611197 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:48.611255 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:48.612279 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:48.612863 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:48.613741 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:48.614097 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:48.614178 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:48.616225 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:48.616229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:48.616232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:48.616263 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:48.618107 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:48.618611 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:48.618616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:48.618618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:48.618752 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:48.619019 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:48.619022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:48.619024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:48.619266 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:48.619370 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:48.619722 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 16:43:48.619878 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:43:48.641129 14150 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:48.642416 14150 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:48.648469 14150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6e61c117fd94d5ea581276ebfbae777: Generated new TSK 0
I20260928 16:43:48.649601 14150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:43:48.664938 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 14073
2026-09-28T16:43:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:48 dist-test-slave-nqgh krb5kdc[14172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:48 dist-test-slave-nqgh krb5kdc[14172](info): set up 2 sockets
Sep 28 16:43:48 dist-test-slave-nqgh krb5kdc[14172](info): commencing operation
krb5kdc: starting...
W20260928 16:43:49.636706 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:43:49 dist-test-slave-nqgh krb5kdc[14172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:43:49.779513 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40579
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:49.894330 14185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:49.894685 14185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:49.894747 14185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:49.899044 14185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:49.899137 14185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:49.899155 14185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:49.899171 14185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:49.899187 14185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:49.904834 14185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:49.906024 14185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:49.907095 14185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:49.916769 14193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:49.916792 14191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:49.916962 14185 server_base.cc:1034] running on GCE node
W20260928 16:43:49.916801 14190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:49.917857 14185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:49.919198 14185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:49.920424 14185 hybrid_clock.cc:648] HybridClock initialized: now 1790613829920407 us; error 47 us; skew 500 ppm
Sep 28 16:43:49 dist-test-slave-nqgh krb5kdc[14172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:49.924381 14185 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:43:49.925618 14185 webserver.cc:460] Webserver started at http://127.0.0.1:41817/ using document root <none> and password file <none>
I20260928 16:43:49.926340 14185 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:49.926421 14185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:49.926605 14185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:49.928861 14185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "71c953042d564c1cbaa824406b3a9a95"
format_stamp: "Formatted at 2026-09-28 16:43:49 on dist-test-slave-nqgh"
server_key: "fd127d92bb986635ac8a335c25a35ad0"
server_key_iv: "4276160a11be31667484cf7edb8ebee2"
server_key_version: "encryptionkey@0"
I20260928 16:43:49.929401 14185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "71c953042d564c1cbaa824406b3a9a95"
format_stamp: "Formatted at 2026-09-28 16:43:49 on dist-test-slave-nqgh"
server_key: "fd127d92bb986635ac8a335c25a35ad0"
server_key_iv: "4276160a11be31667484cf7edb8ebee2"
server_key_version: "encryptionkey@0"
I20260928 16:43:49.933924 14185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:43:49.937227 14200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:49.939043 14185 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 16:43:49.939206 14185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "71c953042d564c1cbaa824406b3a9a95"
format_stamp: "Formatted at 2026-09-28 16:43:49 on dist-test-slave-nqgh"
server_key: "fd127d92bb986635ac8a335c25a35ad0"
server_key_iv: "4276160a11be31667484cf7edb8ebee2"
server_key_version: "encryptionkey@0"
I20260928 16:43:49.939327 14185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:49.960122 14185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:49.963518 14185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:49.963692 14185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:49.972090 14185 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40579
I20260928 16:43:49.972218 14252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40579 every 8 connection(s)
I20260928 16:43:49.973210 14185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:49.976655 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 14185
I20260928 16:43:49.976799 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:43:49.977102 25885 external_mini_cluster.cc:1428] Setting key d73857b891b24c1f86a019760f8970fa
I20260928 16:43:49.977279 14253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:49.986094 14253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95: Bootstrap starting.
I20260928 16:43:49.987668 14256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:49.978786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48953 (local address 127.0.0.1:40579)
0928 16:43:49.979464 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:49.979485 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:49.979535 (+ 50us) server_negotiation.cc:407] Connection header received
0928 16:43:49.980339 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:49.980366 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:49.980786 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:49.981209 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:49.981656 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.982771 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.983772 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.984342 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.984515 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:49.985415 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:49.985420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:49.985439 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:49.985920 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:49.985988 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:49.986014 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:49.986135 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:49.986262 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 16:43:49.986538 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":243,"threads_started":1}
I20260928 16:43:49.989516 14253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:49.990655 14253 log.cc:824] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:49.991991 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:43:49.993708 14253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95: No bootstrap required, opened a new log
I20260928 16:43:49.999471 14256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:49.994451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60037 (local address 127.0.0.1:40579)
0928 16:43:49.994585 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:49.994589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:49.994821 (+ 232us) server_negotiation.cc:407] Connection header received
0928 16:43:49.994996 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:49.994999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:49.995082 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:49.995171 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:49.995580 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.996179 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.997417 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.997671 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.997753 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:49.998770 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:49.998773 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:49.998775 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:49.998789 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:49.998885 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:49.998904 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:49.998982 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:49.999139 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 16:43:49.999203 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:43:49.999298 14253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c953042d564c1cbaa824406b3a9a95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40579 } }
I20260928 16:43:49.999720 14253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:49.999789 14253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c953042d564c1cbaa824406b3a9a95, State: Initialized, Role: FOLLOWER
I20260928 16:43:50.000355 14253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c953042d564c1cbaa824406b3a9a95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40579 } }
I20260928 16:43:50.000512 14253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:50.000583 14253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:50.000712 14253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:50.001938 14253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c953042d564c1cbaa824406b3a9a95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40579 } }
I20260928 16:43:50.002367 14253 leader_election.cc:302] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71c953042d564c1cbaa824406b3a9a95; no voters:
I20260928 16:43:50.002791 14253 leader_election.cc:288] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:50.002977 14263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:50.003283 14263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95 [term 1 LEADER]: Becoming Leader. State: Replica: 71c953042d564c1cbaa824406b3a9a95, State: Running, Role: LEADER
I20260928 16:43:50.003708 14263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c953042d564c1cbaa824406b3a9a95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40579 } }
I20260928 16:43:50.004417 14253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:50.005584 14265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71c953042d564c1cbaa824406b3a9a95. Latest consensus state: current_term: 1 leader_uuid: "71c953042d564c1cbaa824406b3a9a95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c953042d564c1cbaa824406b3a9a95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40579 } } }
I20260928 16:43:50.005760 14265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:50.006101 14264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71c953042d564c1cbaa824406b3a9a95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c953042d564c1cbaa824406b3a9a95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40579 } } }
I20260928 16:43:50.006210 14264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:50.006546 14272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:50.010030 14272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:50.017716 14272 catalog_manager.cc:1347] Generated new cluster ID: 3f1313f61f4c439e9a7598f662515d55
I20260928 16:43:50.017863 14272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:50.038726 14272 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:50.039909 14272 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:50.050763 14272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71c953042d564c1cbaa824406b3a9a95: Generated new TSK 0
I20260928 16:43:50.051551 14272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:43:50.065925 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 14185
2026-09-28T16:43:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:50 dist-test-slave-nqgh krb5kdc[14284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:50 dist-test-slave-nqgh krb5kdc[14284](info): set up 2 sockets
Sep 28 16:43:50 dist-test-slave-nqgh krb5kdc[14284](info): commencing operation
krb5kdc: starting...
W20260928 16:43:51.036273 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:43:51 dist-test-slave-nqgh krb5kdc[14284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:43:51.192135 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:51.333431 14297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:51.333832 14297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:51.333881 14297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:51.339049 14297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:51.339195 14297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:51.339217 14297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:51.339237 14297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:51.339252 14297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:51.346398 14297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:51.348362 14297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:51.349951 14297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:51.361610 14302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:51.361538 14305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:51.361552 14303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:51.362437 14297 server_base.cc:1034] running on GCE node
I20260928 16:43:51.363287 14297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:51.364598 14297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:51.365880 14297 hybrid_clock.cc:648] HybridClock initialized: now 1790613831365851 us; error 57 us; skew 500 ppm
Sep 28 16:43:51 dist-test-slave-nqgh krb5kdc[14284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:51.369570 14297 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:43:51.370977 14297 webserver.cc:460] Webserver started at http://127.0.0.1:39591/ using document root <none> and password file <none>
I20260928 16:43:51.371677 14297 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:51.371755 14297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:51.371984 14297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:51.374321 14297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "f0f59bc6ebdf4e799350a0f967aeea8c"
format_stamp: "Formatted at 2026-09-28 16:43:51 on dist-test-slave-nqgh"
server_key: "effeb37723b42d0cfe2f360b89527ffb"
server_key_iv: "608c619d3d41862b59fbf024d5959ade"
server_key_version: "encryptionkey@0"
I20260928 16:43:51.374972 14297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "f0f59bc6ebdf4e799350a0f967aeea8c"
format_stamp: "Formatted at 2026-09-28 16:43:51 on dist-test-slave-nqgh"
server_key: "effeb37723b42d0cfe2f360b89527ffb"
server_key_iv: "608c619d3d41862b59fbf024d5959ade"
server_key_version: "encryptionkey@0"
I20260928 16:43:51.380020 14297 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 16:43:51.383433 14312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:51.385452 14297 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:43:51.385643 14297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "f0f59bc6ebdf4e799350a0f967aeea8c"
format_stamp: "Formatted at 2026-09-28 16:43:51 on dist-test-slave-nqgh"
server_key: "effeb37723b42d0cfe2f360b89527ffb"
server_key_iv: "608c619d3d41862b59fbf024d5959ade"
server_key_version: "encryptionkey@0"
I20260928 16:43:51.385805 14297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:51.391114 14297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:51.394759 14297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:51.394987 14297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:51.402995 14297 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41509
I20260928 16:43:51.403088 14364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41509 every 8 connection(s)
I20260928 16:43:51.404103 14297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:51.408051 14365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:51.411819 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 14297
I20260928 16:43:51.412029 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:43:51.412377 25885 external_mini_cluster.cc:1428] Setting key c5d4995d099e0726d4051c21a37855d1
I20260928 16:43:51.416357 14365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c: Bootstrap starting.
I20260928 16:43:51.419266 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:51.414460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56133 (local address 127.0.0.1:41509)
0928 16:43:51.415212 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:51.415232 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:51.415282 (+ 50us) server_negotiation.cc:407] Connection header received
0928 16:43:51.416151 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:51.416172 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:51.416568 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:51.416944 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:51.417852 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:51.417856 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:51.417869 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:51.417880 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:51.417939 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:51.417961 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:51.418115 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:51.418192 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 16:43:51.418373 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":216,"threads_started":1}
I20260928 16:43:51.419526 14365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:51.420499 14365 log.cc:824] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:51.423334 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:43:51.423671 14365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c: No bootstrap required, opened a new log
I20260928 16:43:51.427714 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:51.425668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53873 (local address 127.0.0.1:41509)
0928 16:43:51.425830 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:51.425836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:51.425863 (+ 27us) server_negotiation.cc:407] Connection header received
0928 16:43:51.426012 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:51.426015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:51.426069 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:51.426205 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:51.427179 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:51.427184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:51.427188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:51.427203 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:51.427247 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:51.427263 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:51.427385 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:51.427476 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 16:43:51.427535 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:43:51.428766 14365 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f59bc6ebdf4e799350a0f967aeea8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41509 } }
I20260928 16:43:51.429137 14365 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:51.429209 14365 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f59bc6ebdf4e799350a0f967aeea8c, State: Initialized, Role: FOLLOWER
I20260928 16:43:51.429760 14365 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f59bc6ebdf4e799350a0f967aeea8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41509 } }
I20260928 16:43:51.429914 14365 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:51.429962 14365 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:51.430052 14365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:51.431391 14365 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f59bc6ebdf4e799350a0f967aeea8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41509 } }
I20260928 16:43:51.431855 14365 leader_election.cc:302] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0f59bc6ebdf4e799350a0f967aeea8c; no voters:
I20260928 16:43:51.432253 14365 leader_election.cc:288] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:51.432901 14375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:51.433249 14375 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c [term 1 LEADER]: Becoming Leader. State: Replica: f0f59bc6ebdf4e799350a0f967aeea8c, State: Running, Role: LEADER
I20260928 16:43:51.433732 14375 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f59bc6ebdf4e799350a0f967aeea8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41509 } }
I20260928 16:43:51.434333 14365 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:51.435874 14376 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0f59bc6ebdf4e799350a0f967aeea8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f59bc6ebdf4e799350a0f967aeea8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41509 } } }
I20260928 16:43:51.436057 14376 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:51.436599 14377 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0f59bc6ebdf4e799350a0f967aeea8c. Latest consensus state: current_term: 1 leader_uuid: "f0f59bc6ebdf4e799350a0f967aeea8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f59bc6ebdf4e799350a0f967aeea8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41509 } } }
I20260928 16:43:51.436723 14377 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:51.436894 14384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:51.440439 14384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:51.447314 14384 catalog_manager.cc:1347] Generated new cluster ID: 3fd3272330b2481bbd08d4b976759d66
I20260928 16:43:51.447453 14384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:51.471239 14384 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:51.471696 14384 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:51.477317 14384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0f59bc6ebdf4e799350a0f967aeea8c: Generated new TSK 0
I20260928 16:43:51.478058 14384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:43:51.488914 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 14297
2026-09-28T16:43:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:51 dist-test-slave-nqgh krb5kdc[14396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:51 dist-test-slave-nqgh krb5kdc[14396](info): set up 2 sockets
Sep 28 16:43:51 dist-test-slave-nqgh krb5kdc[14396](info): commencing operation
krb5kdc: starting...
W20260928 16:43:52.468278 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:43:52 dist-test-slave-nqgh krb5kdc[14396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:43:52.607245 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41511
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:52.744701 14409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:52.745189 14409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:52.745277 14409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:52.751082 14409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:52.751179 14409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:52.751200 14409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:52.751219 14409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:52.751235 14409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:52.757711 14409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:52.759330 14409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:52.760645 14409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:52.771256 14417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:52.771311 14414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:52.771485 14409 server_base.cc:1034] running on GCE node
W20260928 16:43:52.771262 14415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:52.772279 14409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:52.773598 14409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:52.774791 14409 hybrid_clock.cc:648] HybridClock initialized: now 1790613832774758 us; error 58 us; skew 500 ppm
Sep 28 16:43:52 dist-test-slave-nqgh krb5kdc[14396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:52.782294 14409 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:43:52.783567 14409 webserver.cc:460] Webserver started at http://127.0.0.1:35885/ using document root <none> and password file <none>
I20260928 16:43:52.784215 14409 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:52.784289 14409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:52.784498 14409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:52.786787 14409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "a4409415a4fe49da8d5709bbbd6a89ba"
format_stamp: "Formatted at 2026-09-28 16:43:52 on dist-test-slave-nqgh"
server_key: "2d894e69fcd336bd394c6dfca45337ad"
server_key_iv: "1c241c4587cf6e2e888272f0a8e5eeae"
server_key_version: "encryptionkey@0"
I20260928 16:43:52.787429 14409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "a4409415a4fe49da8d5709bbbd6a89ba"
format_stamp: "Formatted at 2026-09-28 16:43:52 on dist-test-slave-nqgh"
server_key: "2d894e69fcd336bd394c6dfca45337ad"
server_key_iv: "1c241c4587cf6e2e888272f0a8e5eeae"
server_key_version: "encryptionkey@0"
I20260928 16:43:52.792098 14409 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:43:52.795645 14424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:52.797353 14409 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:43:52.797559 14409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "a4409415a4fe49da8d5709bbbd6a89ba"
format_stamp: "Formatted at 2026-09-28 16:43:52 on dist-test-slave-nqgh"
server_key: "2d894e69fcd336bd394c6dfca45337ad"
server_key_iv: "1c241c4587cf6e2e888272f0a8e5eeae"
server_key_version: "encryptionkey@0"
I20260928 16:43:52.797694 14409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:52.817662 14409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:52.821276 14409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:52.821506 14409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:52.829579 14409 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41511
I20260928 16:43:52.829870 14476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41511 every 8 connection(s)
I20260928 16:43:52.830798 14409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:52.834858 14477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:52.836634 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 14409
I20260928 16:43:52.836772 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:43:52.837128 25885 external_mini_cluster.cc:1428] Setting key 07a36443d6f91c97136647d68e791d87
I20260928 16:43:52.843572 14477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba: Bootstrap starting.
I20260928 16:43:52.847857 14477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba: Neither blocks nor log segments found. Creating new log.
Sep 28 16:43:52 dist-test-slave-nqgh krb5kdc[14396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 16:43:52.849023 14477 log.cc:824] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:52.852406 14477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba: No bootstrap required, opened a new log
I20260928 16:43:52.857126 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:52.839060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:49275 (local address 127.0.0.1:41511)
0928 16:43:52.839575 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:52.839591 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:52.839631 (+ 40us) server_negotiation.cc:407] Connection header received
0928 16:43:52.840529 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:52.840552 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:52.840974 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:52.841291 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:52.842544 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:52.843747 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:52.845120 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:52.845691 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:52.845847 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:52.849840 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:52.849849 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:52.849871 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:52.849912 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:52.852196 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:52.853274 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:52.853282 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:52.853291 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:52.853491 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:52.853993 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:52.853996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:52.853998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:52.854715 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:52.855018 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:52.855380 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 16:43:52.855744 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":174,"threads_started":1}
I20260928 16:43:52.858292 14477 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4409415a4fe49da8d5709bbbd6a89ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41511 } }
I20260928 16:43:52.858750 14477 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:52.858896 14477 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4409415a4fe49da8d5709bbbd6a89ba, State: Initialized, Role: FOLLOWER
I20260928 16:43:52.859459 14477 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4409415a4fe49da8d5709bbbd6a89ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41511 } }
I20260928 16:43:52.859628 14477 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:52.859702 14477 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:52.859815 14477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:52.861306 14477 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4409415a4fe49da8d5709bbbd6a89ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41511 } }
I20260928 16:43:52.861656 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:43:52.861785 14477 leader_election.cc:302] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4409415a4fe49da8d5709bbbd6a89ba; no voters:
I20260928 16:43:52.862244 14477 leader_election.cc:288] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:52.862462 14482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:52.862918 14482 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba [term 1 LEADER]: Becoming Leader. State: Replica: a4409415a4fe49da8d5709bbbd6a89ba, State: Running, Role: LEADER
I20260928 16:43:52.863355 14482 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4409415a4fe49da8d5709bbbd6a89ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41511 } }
I20260928 16:43:52.864143 14477 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:52.865392 14484 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4409415a4fe49da8d5709bbbd6a89ba. Latest consensus state: current_term: 1 leader_uuid: "a4409415a4fe49da8d5709bbbd6a89ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4409415a4fe49da8d5709bbbd6a89ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41511 } } }
I20260928 16:43:52.865594 14484 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:52.865391 14483 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4409415a4fe49da8d5709bbbd6a89ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4409415a4fe49da8d5709bbbd6a89ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41511 } } }
I20260928 16:43:52.865837 14483 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:52.866220 14496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:52.871308 14496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:52.873903 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:52.864764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:51463 (local address 127.0.0.1:41511)
0928 16:43:52.864923 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:52.864930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:52.865535 (+ 605us) server_negotiation.cc:407] Connection header received
0928 16:43:52.865757 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:52.865761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:52.865819 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:52.865876 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:52.867054 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:52.867770 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:52.868828 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:52.869107 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:52.869206 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:52.870700 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:52.870704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:52.870707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:52.870735 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:52.872137 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:52.872615 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:52.872618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:52.872621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:52.872695 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:52.872995 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:52.872998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:52.873000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:52.873197 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:52.873329 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:52.873608 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 16:43:52.873712 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:43:52.879619 14496 catalog_manager.cc:1347] Generated new cluster ID: 454d3e18f59d49a7ae402cf30c45a458
I20260928 16:43:52.879757 14496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:52.896654 14496 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:52.898329 14496 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:52.906363 14496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4409415a4fe49da8d5709bbbd6a89ba: Generated new TSK 0
I20260928 16:43:52.907670 14496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:43:52.917385 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 14409
W20260928 16:43:52.923005 14394 connection.cc:535] client connection to 127.0.0.1:41511 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41511): Connection reset by peer (error 104)
2026-09-28T16:43:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:52 dist-test-slave-nqgh krb5kdc[14508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:52 dist-test-slave-nqgh krb5kdc[14508](info): set up 2 sockets
Sep 28 16:43:52 dist-test-slave-nqgh krb5kdc[14508](info): commencing operation
krb5kdc: starting...
W20260928 16:43:53.896663 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:43:53 dist-test-slave-nqgh krb5kdc[14508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:43:54.039996 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38529
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:54.179481 14521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:54.179906 14521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:54.179956 14521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:54.184741 14521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:54.184865 14521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:54.184886 14521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:54.184904 14521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:54.184919 14521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:54.191820 14521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:54.193518 14521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:54.194991 14521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:54.206458 14529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:54.206458 14527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:54.206449 14526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:54.206866 14521 server_base.cc:1034] running on GCE node
I20260928 16:43:54.207835 14521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:54.209260 14521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:54.210506 14521 hybrid_clock.cc:648] HybridClock initialized: now 1790613834210482 us; error 49 us; skew 500 ppm
Sep 28 16:43:54 dist-test-slave-nqgh krb5kdc[14508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:54.213639 14521 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:43:54.214768 14521 webserver.cc:460] Webserver started at http://127.0.0.1:45333/ using document root <none> and password file <none>
I20260928 16:43:54.215385 14521 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:54.215454 14521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:54.215648 14521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:54.217948 14521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "d985c32314504fdda4a3daa837ca4618"
format_stamp: "Formatted at 2026-09-28 16:43:54 on dist-test-slave-nqgh"
server_key: "c024630871cd32a92ed5bff921eddb7c"
server_key_iv: "0657945416df881e61a7cd5689051b3d"
server_key_version: "encryptionkey@0"
I20260928 16:43:54.218523 14521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "d985c32314504fdda4a3daa837ca4618"
format_stamp: "Formatted at 2026-09-28 16:43:54 on dist-test-slave-nqgh"
server_key: "c024630871cd32a92ed5bff921eddb7c"
server_key_iv: "0657945416df881e61a7cd5689051b3d"
server_key_version: "encryptionkey@0"
I20260928 16:43:54.223106 14521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 16:43:54.226400 14536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:54.228374 14521 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:43:54.228533 14521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "d985c32314504fdda4a3daa837ca4618"
format_stamp: "Formatted at 2026-09-28 16:43:54 on dist-test-slave-nqgh"
server_key: "c024630871cd32a92ed5bff921eddb7c"
server_key_iv: "0657945416df881e61a7cd5689051b3d"
server_key_version: "encryptionkey@0"
I20260928 16:43:54.228659 14521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:54.251555 14521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:54.255189 14521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:54.255375 14521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:54.264144 14521 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38529
I20260928 16:43:54.264207 14588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38529 every 8 connection(s)
I20260928 16:43:54.265277 14521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:54.269078 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 14521
I20260928 16:43:54.269208 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:43:54.269536 25885 external_mini_cluster.cc:1428] Setting key ea0e49225be7188304ff95d30bc7f156
I20260928 16:43:54.269560 14589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:54.279367 14589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618: Bootstrap starting.
I20260928 16:43:54.281864 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:54.271204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:34485 (local address 127.0.0.1:38529)
0928 16:43:54.271714 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:54.271733 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:54.271775 (+ 42us) server_negotiation.cc:407] Connection header received
0928 16:43:54.272566 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:54.272588 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:54.273007 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:54.273354 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:54.274110 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:54.275378 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:54.276903 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:54.277656 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:54.277835 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:54.279402 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:54.279410 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:54.279430 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:54.280096 (+ 666us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:54.280186 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:54.280220 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:54.280382 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:54.280681 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 16:43:54.280865 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":139,"threads_started":1}
I20260928 16:43:54.283006 14589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:54.284039 14589 log.cc:824] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:54.286374 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:43:54.287014 14589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618: No bootstrap required, opened a new log
I20260928 16:43:54.292588 14589 raft_consensus.cc:348] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d985c32314504fdda4a3daa837ca4618" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38529 } }
I20260928 16:43:54.293081 14589 raft_consensus.cc:374] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:54.293190 14589 raft_consensus.cc:729] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d985c32314504fdda4a3daa837ca4618, State: Initialized, Role: FOLLOWER
I20260928 16:43:54.293970 14589 consensus_queue.cc:260] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d985c32314504fdda4a3daa837ca4618" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38529 } }
I20260928 16:43:54.294232 14589 raft_consensus.cc:388] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:54.294405 14589 raft_consensus.cc:482] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:54.294555 14589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:54.294726 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:54.289308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:56781 (local address 127.0.0.1:38529)
0928 16:43:54.289521 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:54.289525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:54.289734 (+ 209us) server_negotiation.cc:407] Connection header received
0928 16:43:54.289926 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:54.289930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:54.289990 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:54.290080 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:54.290467 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:54.291054 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:54.292317 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:54.292571 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:54.292671 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:54.293901 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:54.293905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:54.293907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:54.293923 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:54.293958 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:54.293971 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:54.294071 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:54.294420 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 16:43:54.294473 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260928 16:43:54.296343 14589 raft_consensus.cc:504] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d985c32314504fdda4a3daa837ca4618" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38529 } }
I20260928 16:43:54.296912 14589 leader_election.cc:302] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d985c32314504fdda4a3daa837ca4618; no voters:
I20260928 16:43:54.297401 14589 leader_election.cc:288] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:54.297695 14599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:54.298097 14599 raft_consensus.cc:686] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618 [term 1 LEADER]: Becoming Leader. State: Replica: d985c32314504fdda4a3daa837ca4618, State: Running, Role: LEADER
I20260928 16:43:54.298502 14599 consensus_queue.cc:237] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d985c32314504fdda4a3daa837ca4618" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38529 } }
I20260928 16:43:54.299213 14589 sys_catalog.cc:564] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:54.300696 14600 sys_catalog.cc:455] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d985c32314504fdda4a3daa837ca4618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d985c32314504fdda4a3daa837ca4618" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38529 } } }
I20260928 16:43:54.300889 14600 sys_catalog.cc:458] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:54.300655 14601 sys_catalog.cc:455] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d985c32314504fdda4a3daa837ca4618. Latest consensus state: current_term: 1 leader_uuid: "d985c32314504fdda4a3daa837ca4618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d985c32314504fdda4a3daa837ca4618" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38529 } } }
I20260928 16:43:54.301232 14601 sys_catalog.cc:458] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:54.302176 14609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:54.305493 14609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:54.311352 14609 catalog_manager.cc:1347] Generated new cluster ID: 968dc3ca7bda49059525a589a298338f
I20260928 16:43:54.311450 14609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:54.332114 14609 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:54.333405 14609 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:54.349476 14609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d985c32314504fdda4a3daa837ca4618: Generated new TSK 0
I20260928 16:43:54.350306 14609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:43:54.356961 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 14521
W20260928 16:43:54.363904 14506 connection.cc:535] client connection to 127.0.0.1:38529 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38529): Connection reset by peer (error 104)
2026-09-28T16:43:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:54 dist-test-slave-nqgh krb5kdc[14620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:54 dist-test-slave-nqgh krb5kdc[14620](info): set up 2 sockets
Sep 28 16:43:54 dist-test-slave-nqgh krb5kdc[14620](info): commencing operation
krb5kdc: starting...
W20260928 16:43:55.336745 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:43:55 dist-test-slave-nqgh krb5kdc[14620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:43:55.479118 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44715
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:55.613481 14633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:55.613890 14633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:55.613957 14633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:55.618467 14633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:55.618582 14633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:55.618610 14633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:55.618657 14633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:55.618681 14633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:55.625005 14633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:55.626562 14633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:55.627946 14633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:55.638353 14638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:55.638371 14641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:55.638371 14639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:55.638701 14633 server_base.cc:1034] running on GCE node
I20260928 16:43:55.639789 14633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:55.641319 14633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:55.642535 14633 hybrid_clock.cc:648] HybridClock initialized: now 1790613835642517 us; error 39 us; skew 500 ppm
Sep 28 16:43:55 dist-test-slave-nqgh krb5kdc[14620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:55.646268 14633 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:43:55.647710 14633 webserver.cc:460] Webserver started at http://127.0.0.1:38803/ using document root <none> and password file <none>
I20260928 16:43:55.648478 14633 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:55.648578 14633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:55.648967 14633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:55.651978 14633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "467589d9d5bf43a98adb9fd53e2991d4"
format_stamp: "Formatted at 2026-09-28 16:43:55 on dist-test-slave-nqgh"
server_key: "782785415819c167c0888429c37df09d"
server_key_iv: "9ae84e11e7a6cdaf7f60ec206247b1e6"
server_key_version: "encryptionkey@0"
I20260928 16:43:55.652737 14633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "467589d9d5bf43a98adb9fd53e2991d4"
format_stamp: "Formatted at 2026-09-28 16:43:55 on dist-test-slave-nqgh"
server_key: "782785415819c167c0888429c37df09d"
server_key_iv: "9ae84e11e7a6cdaf7f60ec206247b1e6"
server_key_version: "encryptionkey@0"
I20260928 16:43:55.658161 14633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:43:55.661644 14648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:55.663681 14633 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:43:55.663880 14633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "467589d9d5bf43a98adb9fd53e2991d4"
format_stamp: "Formatted at 2026-09-28 16:43:55 on dist-test-slave-nqgh"
server_key: "782785415819c167c0888429c37df09d"
server_key_iv: "9ae84e11e7a6cdaf7f60ec206247b1e6"
server_key_version: "encryptionkey@0"
I20260928 16:43:55.664057 14633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:55.670085 14633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:55.673990 14633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:55.674222 14633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:55.683378 14633 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44715
I20260928 16:43:55.683401 14700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44715 every 8 connection(s)
I20260928 16:43:55.684799 14633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:55.687834 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 14633
I20260928 16:43:55.687991 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:43:55.688330 25885 external_mini_cluster.cc:1428] Setting key 520daf6b7233eb4deaa2ae03e957dab7
I20260928 16:43:55.688334 14701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:55.695420 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:55.690325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:38651 (local address 127.0.0.1:44715)
0928 16:43:55.690947 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:55.690968 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:55.691015 (+ 47us) server_negotiation.cc:407] Connection header received
0928 16:43:55.691944 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:55.691968 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:55.692353 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:55.692756 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:55.693791 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:55.693797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:55.693811 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:55.693824 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:55.693891 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:55.693912 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:55.694049 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:55.694158 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 16:43:55.694361 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":188,"threads_started":1}
I20260928 16:43:55.696156 14701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4: Bootstrap starting.
I20260928 16:43:55.698964 14701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:55.699295 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:43:55.699927 14701 log.cc:824] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:55.703140 14701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4: No bootstrap required, opened a new log
I20260928 16:43:55.704185 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:55.702016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:35081 (local address 127.0.0.1:44715)
0928 16:43:55.702198 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:55.702204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:55.702446 (+ 242us) server_negotiation.cc:407] Connection header received
0928 16:43:55.702593 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:55.702597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:55.702673 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:55.702774 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:55.703677 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:55.703680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:55.703683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:55.703696 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:55.703731 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:55.703746 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:55.703862 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:55.703999 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 16:43:55.704049 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:43:55.707389 14701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467589d9d5bf43a98adb9fd53e2991d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44715 } }
I20260928 16:43:55.707667 14701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:55.707703 14701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 467589d9d5bf43a98adb9fd53e2991d4, State: Initialized, Role: FOLLOWER
I20260928 16:43:55.708117 14701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467589d9d5bf43a98adb9fd53e2991d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44715 } }
I20260928 16:43:55.708243 14701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:55.708287 14701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:55.708365 14701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:55.709407 14701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467589d9d5bf43a98adb9fd53e2991d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44715 } }
I20260928 16:43:55.709908 14701 leader_election.cc:302] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 467589d9d5bf43a98adb9fd53e2991d4; no voters:
I20260928 16:43:55.710306 14701 leader_election.cc:288] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:55.710599 14711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:55.710978 14711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4 [term 1 LEADER]: Becoming Leader. State: Replica: 467589d9d5bf43a98adb9fd53e2991d4, State: Running, Role: LEADER
I20260928 16:43:55.711431 14711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467589d9d5bf43a98adb9fd53e2991d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44715 } }
I20260928 16:43:55.711926 14701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:55.713616 14713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 467589d9d5bf43a98adb9fd53e2991d4. Latest consensus state: current_term: 1 leader_uuid: "467589d9d5bf43a98adb9fd53e2991d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467589d9d5bf43a98adb9fd53e2991d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44715 } } }
I20260928 16:43:55.713825 14713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:55.714188 14712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "467589d9d5bf43a98adb9fd53e2991d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467589d9d5bf43a98adb9fd53e2991d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44715 } } }
I20260928 16:43:55.714293 14712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:55.714392 14723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:55.718370 14723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:55.725703 14723 catalog_manager.cc:1347] Generated new cluster ID: f95d6dbb071f40ca964e2062fc889fb5
I20260928 16:43:55.725847 14723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:55.747448 14723 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:55.747956 14723 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:55.757692 14723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 467589d9d5bf43a98adb9fd53e2991d4: Generated new TSK 0
I20260928 16:43:55.758682 14723 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:43:55.767851 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 14633
2026-09-28T16:43:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:55 dist-test-slave-nqgh krb5kdc[14732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:55 dist-test-slave-nqgh krb5kdc[14732](info): set up 2 sockets
Sep 28 16:43:55 dist-test-slave-nqgh krb5kdc[14732](info): commencing operation
krb5kdc: starting...
W20260928 16:43:56.744740 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:43:56 dist-test-slave-nqgh krb5kdc[14732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:43:56.884204 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46123
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:57.024307 14745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:57.024724 14745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:57.024797 14745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:57.029422 14745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:57.029543 14745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:57.029575 14745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:57.029598 14745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:57.029619 14745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:57.036139 14745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46123
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:57.037717 14745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:57.039163 14745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:57.050014 14753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:57.050109 14750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:57.050351 14745 server_base.cc:1034] running on GCE node
W20260928 16:43:57.050014 14751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:57.051337 14745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:57.052878 14745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:57.054078 14745 hybrid_clock.cc:648] HybridClock initialized: now 1790613837054051 us; error 55 us; skew 500 ppm
Sep 28 16:43:57 dist-test-slave-nqgh krb5kdc[14732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:57.057857 14745 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:43:57.059255 14745 webserver.cc:460] Webserver started at http://127.0.0.1:35999/ using document root <none> and password file <none>
I20260928 16:43:57.060009 14745 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:57.060092 14745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:57.060348 14745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:57.063162 14745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "5d7de5507fcf47db9498c8cc66f21ca7"
format_stamp: "Formatted at 2026-09-28 16:43:57 on dist-test-slave-nqgh"
server_key: "6b2c561d3bc406afa238f6ec81b23aa6"
server_key_iv: "a0076df64f7ed3dc0c08c5a6b6db59dd"
server_key_version: "encryptionkey@0"
I20260928 16:43:57.063887 14745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "5d7de5507fcf47db9498c8cc66f21ca7"
format_stamp: "Formatted at 2026-09-28 16:43:57 on dist-test-slave-nqgh"
server_key: "6b2c561d3bc406afa238f6ec81b23aa6"
server_key_iv: "a0076df64f7ed3dc0c08c5a6b6db59dd"
server_key_version: "encryptionkey@0"
I20260928 16:43:57.069188 14745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 16:43:57.073020 14760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:57.074877 14745 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:43:57.075075 14745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "5d7de5507fcf47db9498c8cc66f21ca7"
format_stamp: "Formatted at 2026-09-28 16:43:57 on dist-test-slave-nqgh"
server_key: "6b2c561d3bc406afa238f6ec81b23aa6"
server_key_iv: "a0076df64f7ed3dc0c08c5a6b6db59dd"
server_key_version: "encryptionkey@0"
I20260928 16:43:57.075227 14745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:57.090903 14745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:57.094990 14745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:57.095221 14745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:57.105090 14745 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46123
I20260928 16:43:57.105175 14812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46123 every 8 connection(s)
I20260928 16:43:57.106469 14745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:57.110039 14813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:57.113011 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 14745
I20260928 16:43:57.113166 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:43:57.113471 25885 external_mini_cluster.cc:1428] Setting key 41067c3711ee2c858812dcc6ab98108c
I20260928 16:43:57.116951 14813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7: Bootstrap starting.
I20260928 16:43:57.120138 14813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:57.121058 14813 log.cc:824] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:57.124115 14813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7: No bootstrap required, opened a new log
Sep 28 16:43:57 dist-test-slave-nqgh krb5kdc[14732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 16:43:57.128053 14813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7de5507fcf47db9498c8cc66f21ca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46123 } }
I20260928 16:43:57.128383 14813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:57.128440 14813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d7de5507fcf47db9498c8cc66f21ca7, State: Initialized, Role: FOLLOWER
I20260928 16:43:57.128939 14813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7de5507fcf47db9498c8cc66f21ca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46123 } }
I20260928 16:43:57.129097 14813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:57.129170 14813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:57.129272 14813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:57.130369 14813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7de5507fcf47db9498c8cc66f21ca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46123 } }
I20260928 16:43:57.130784 14813 leader_election.cc:302] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d7de5507fcf47db9498c8cc66f21ca7; no voters:
I20260928 16:43:57.131148 14813 leader_election.cc:288] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:57.131255 14818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:57.131518 14818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7 [term 1 LEADER]: Becoming Leader. State: Replica: 5d7de5507fcf47db9498c8cc66f21ca7, State: Running, Role: LEADER
I20260928 16:43:57.131909 14818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7de5507fcf47db9498c8cc66f21ca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46123 } }
I20260928 16:43:57.132820 14813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:57.133044 14816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:57.115456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:40027 (local address 127.0.0.1:46123)
0928 16:43:57.116054 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:57.116074 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:57.116126 (+ 52us) server_negotiation.cc:407] Connection header received
0928 16:43:57.117002 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:57.117027 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:57.117579 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:57.117991 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:57.119289 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:57.120625 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:57.122030 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:57.123042 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:57.123282 (+ 240us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:57.126426 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:57.126434 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:57.126458 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:57.126504 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:57.129204 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:57.129881 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:57.129888 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:57.129895 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:57.130031 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:57.130361 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:57.130364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:57.130365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:57.131016 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:57.131310 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:57.131578 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 16:43:57.131903 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":170,"threads_started":1}
I20260928 16:43:57.133260 14818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d7de5507fcf47db9498c8cc66f21ca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7de5507fcf47db9498c8cc66f21ca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46123 } } }
I20260928 16:43:57.133523 14818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:57.133771 14818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d7de5507fcf47db9498c8cc66f21ca7. Latest consensus state: current_term: 1 leader_uuid: "5d7de5507fcf47db9498c8cc66f21ca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7de5507fcf47db9498c8cc66f21ca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46123 } } }
I20260928 16:43:57.133953 14818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:57.137712 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 16:43:57.139526 14833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:43:57.139670 14833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:43:57.139818 14823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:57.141525 14823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:57.151024 14823 catalog_manager.cc:1347] Generated new cluster ID: f3f6b9ae6f8e4e11b7c24e1a37df5458
I20260928 16:43:57.151121 14823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:57.151257 14816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:57.140778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:55861 (local address 127.0.0.1:46123)
0928 16:43:57.140945 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:57.140950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:57.141319 (+ 369us) server_negotiation.cc:407] Connection header received
0928 16:43:57.141521 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:57.141524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:57.141604 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:57.141714 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:57.142807 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:57.143709 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:57.144884 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:57.145164 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:57.145270 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:57.146807 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:57.146811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:57.146813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:57.146842 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:57.148337 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:57.149278 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:57.149284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:57.149286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:57.149423 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:57.149894 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:57.149898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:57.149901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:57.150309 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:57.150499 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:57.150952 (+ 453us) server_negotiation.cc:299] Negotiation successful
0928 16:43:57.151100 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:43:57.174151 14823 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:57.175315 14823 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:57.181346 14823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d7de5507fcf47db9498c8cc66f21ca7: Generated new TSK 0
I20260928 16:43:57.182271 14823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:43:57.194813 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 14745
W20260928 16:43:57.201797 14730 connection.cc:535] client connection to 127.0.0.1:46123 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46123): Connection reset by peer (error 104)
2026-09-28T16:43:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:57 dist-test-slave-nqgh krb5kdc[14844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:57 dist-test-slave-nqgh krb5kdc[14844](info): set up 2 sockets
Sep 28 16:43:57 dist-test-slave-nqgh krb5kdc[14844](info): commencing operation
krb5kdc: starting...
W20260928 16:43:58.168947 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:43:58 dist-test-slave-nqgh krb5kdc[14844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:43:58.314548 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:58.460106 14857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:58.460536 14857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:58.460587 14857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:58.465406 14857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:58.465533 14857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:58.465554 14857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:58.465571 14857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:58.465587 14857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:58.472354 14857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34427
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:58.474016 14857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:58.475476 14857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:58.486843 14863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:58.486843 14862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:58.486975 14865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:58.487150 14857 server_base.cc:1034] running on GCE node
I20260928 16:43:58.487928 14857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:58.489244 14857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:58.490453 14857 hybrid_clock.cc:648] HybridClock initialized: now 1790613838490445 us; error 48 us; skew 500 ppm
Sep 28 16:43:58 dist-test-slave-nqgh krb5kdc[14844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:58.499101 14857 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:43:58.500360 14857 webserver.cc:460] Webserver started at http://127.0.0.1:46797/ using document root <none> and password file <none>
I20260928 16:43:58.501052 14857 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:58.501125 14857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:58.501340 14857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:58.503839 14857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "349a8c09cab04442830b848f86efd401"
format_stamp: "Formatted at 2026-09-28 16:43:58 on dist-test-slave-nqgh"
server_key: "518e4ce1edeeea7b648430515cd2b159"
server_key_iv: "8f57772ca6b003bf61f64e29666c9129"
server_key_version: "encryptionkey@0"
I20260928 16:43:58.504522 14857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "349a8c09cab04442830b848f86efd401"
format_stamp: "Formatted at 2026-09-28 16:43:58 on dist-test-slave-nqgh"
server_key: "518e4ce1edeeea7b648430515cd2b159"
server_key_iv: "8f57772ca6b003bf61f64e29666c9129"
server_key_version: "encryptionkey@0"
I20260928 16:43:58.509291 14857 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:43:58.512843 14872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:58.514843 14857 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:43:58.515018 14857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "349a8c09cab04442830b848f86efd401"
format_stamp: "Formatted at 2026-09-28 16:43:58 on dist-test-slave-nqgh"
server_key: "518e4ce1edeeea7b648430515cd2b159"
server_key_iv: "8f57772ca6b003bf61f64e29666c9129"
server_key_version: "encryptionkey@0"
I20260928 16:43:58.515151 14857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:58.536264 14857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:58.540483 14857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:58.540678 14857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:58.549326 14857 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34427
I20260928 16:43:58.549383 14924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34427 every 8 connection(s)
I20260928 16:43:58.550537 14857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:58.554179 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 14857
I20260928 16:43:58.554316 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:43:58.554598 25885 external_mini_cluster.cc:1428] Setting key 7ba466cbc7c4c0514eae1a7b76f89b73
I20260928 16:43:58.554855 14925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:58.564075 14925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401: Bootstrap starting.
I20260928 16:43:58.566046 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:58.556280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:54079 (local address 127.0.0.1:34427)
0928 16:43:58.556785 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:58.556802 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:58.556854 (+ 52us) server_negotiation.cc:407] Connection header received
0928 16:43:58.557580 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:58.557599 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:58.558006 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:58.558324 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:58.559000 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:58.560096 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:58.561692 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:58.562424 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:58.562583 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:58.563798 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:58.563804 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:58.563822 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:58.564315 (+ 493us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:58.564395 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:58.564422 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:58.564554 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:58.564951 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 16:43:58.565143 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":143,"threads_started":1}
I20260928 16:43:58.567631 14925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:58.568699 14925 log.cc:824] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:58.570534 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:43:58.571890 14925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401: No bootstrap required, opened a new log
I20260928 16:43:58.576894 14925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349a8c09cab04442830b848f86efd401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34427 } }
I20260928 16:43:58.577356 14925 raft_consensus.cc:374] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:58.577414 14925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349a8c09cab04442830b848f86efd401, State: Initialized, Role: FOLLOWER
I20260928 16:43:58.578110 14925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349a8c09cab04442830b848f86efd401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34427 } }
I20260928 16:43:58.578414 14925 raft_consensus.cc:388] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:58.578514 14925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:58.578698 14925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:58.579053 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:58.572857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:47637 (local address 127.0.0.1:34427)
0928 16:43:58.573009 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:58.573014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:58.573111 (+ 97us) server_negotiation.cc:407] Connection header received
0928 16:43:58.573233 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:58.573237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:58.573306 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:58.573429 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:58.573727 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:58.575377 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:58.576466 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:58.576854 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:58.576993 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:58.578175 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:58.578179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:58.578183 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:58.578203 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:58.578249 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:58.578266 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:58.578388 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:58.578662 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 16:43:58.578731 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:43:58.580533 14925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349a8c09cab04442830b848f86efd401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34427 } }
I20260928 16:43:58.581115 14925 leader_election.cc:302] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 349a8c09cab04442830b848f86efd401; no voters:
I20260928 16:43:58.581540 14925 leader_election.cc:288] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:58.581887 14935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:58.582224 14935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401 [term 1 LEADER]: Becoming Leader. State: Replica: 349a8c09cab04442830b848f86efd401, State: Running, Role: LEADER
I20260928 16:43:58.582695 14935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349a8c09cab04442830b848f86efd401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34427 } }
I20260928 16:43:58.583321 14925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:58.585371 14936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "349a8c09cab04442830b848f86efd401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349a8c09cab04442830b848f86efd401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34427 } } }
I20260928 16:43:58.585420 14937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 349a8c09cab04442830b848f86efd401. Latest consensus state: current_term: 1 leader_uuid: "349a8c09cab04442830b848f86efd401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349a8c09cab04442830b848f86efd401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34427 } } }
I20260928 16:43:58.585592 14936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:58.585600 14937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401 [sys.catalog]: This master's current role is: LEADER
W20260928 16:43:58.589846 14949 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:43:58.589998 14949 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:43:58.590075 14950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:58.591231 14950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:58.598363 14950 catalog_manager.cc:1347] Generated new cluster ID: 71100a78b8314e77a77f2096c9079601
I20260928 16:43:58.598529 14950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:58.613160 14950 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:58.614912 14950 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:58.627540 14950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 349a8c09cab04442830b848f86efd401: Generated new TSK 0
I20260928 16:43:58.628821 14950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:43:58.641378 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 14857
W20260928 16:43:58.647745 14842 connection.cc:535] client connection to 127.0.0.1:34427 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34427): Connection reset by peer (error 104)
2026-09-28T16:43:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:58 dist-test-slave-nqgh krb5kdc[14956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:58 dist-test-slave-nqgh krb5kdc[14956](info): set up 2 sockets
Sep 28 16:43:58 dist-test-slave-nqgh krb5kdc[14956](info): commencing operation
krb5kdc: starting...
W20260928 16:43:59.620435 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:43:59 dist-test-slave-nqgh krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:43:59.759217 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39233
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:59.896914 14969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:59.897325 14969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:59.897377 14969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:59.901844 14969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:59.901945 14969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:59.901966 14969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:59.901983 14969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:59.901998 14969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:59.908517 14969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39233
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:43:59.910075 14969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:59.911660 14969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:59.922320 14974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:59.922392 14977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:59.922392 14975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:59.922827 14969 server_base.cc:1034] running on GCE node
I20260928 16:43:59.923538 14969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:59.924851 14969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:59.926069 14969 hybrid_clock.cc:648] HybridClock initialized: now 1790613839926044 us; error 58 us; skew 500 ppm
Sep 28 16:43:59 dist-test-slave-nqgh krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:59.929489 14969 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:43:59.930565 14969 webserver.cc:460] Webserver started at http://127.0.0.1:35963/ using document root <none> and password file <none>
I20260928 16:43:59.931265 14969 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:59.931342 14969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:59.931557 14969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:59.933905 14969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "786d320bd4484487b59908b8058a3ccc"
format_stamp: "Formatted at 2026-09-28 16:43:59 on dist-test-slave-nqgh"
server_key: "136a431bf35a5497d23c0882e4d812f0"
server_key_iv: "776c7483ee8ed0a5bae104767d55ba88"
server_key_version: "encryptionkey@0"
I20260928 16:43:59.934520 14969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "786d320bd4484487b59908b8058a3ccc"
format_stamp: "Formatted at 2026-09-28 16:43:59 on dist-test-slave-nqgh"
server_key: "136a431bf35a5497d23c0882e4d812f0"
server_key_iv: "776c7483ee8ed0a5bae104767d55ba88"
server_key_version: "encryptionkey@0"
I20260928 16:43:59.939049 14969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:43:59.942361 14984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:59.944484 14969 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:43:59.944679 14969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "786d320bd4484487b59908b8058a3ccc"
format_stamp: "Formatted at 2026-09-28 16:43:59 on dist-test-slave-nqgh"
server_key: "136a431bf35a5497d23c0882e4d812f0"
server_key_iv: "776c7483ee8ed0a5bae104767d55ba88"
server_key_version: "encryptionkey@0"
I20260928 16:43:59.944849 14969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:59.950176 14969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:59.954336 14969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:59.954568 14969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:59.964442 14969 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39233
I20260928 16:43:59.964466 15036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39233 every 8 connection(s)
I20260928 16:43:59.965943 14969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:59.967813 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 14969
I20260928 16:43:59.967983 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:43:59.968406 25885 external_mini_cluster.cc:1428] Setting key 39406931d9707ebdf81622a8cef238da
I20260928 16:43:59.970496 15037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:59.975819 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:59.970204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:59309 (local address 127.0.0.1:39233)
0928 16:43:59.970744 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:59.970769 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:59.970819 (+ 50us) server_negotiation.cc:407] Connection header received
0928 16:43:59.971551 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:59.971572 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:59.971929 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:59.972274 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:59.973464 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:59.973471 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:59.973489 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:59.973506 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:59.973591 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:59.973622 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:59.973797 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:59.974162 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 16:43:59.974489 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":149,"threads_started":1}
I20260928 16:43:59.979828 15037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc: Bootstrap starting.
I20260928 16:43:59.980379 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:43:59.982985 15037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:59.984077 15037 log.cc:824] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:59.984853 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:59.982538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:42453 (local address 127.0.0.1:39233)
0928 16:43:59.982728 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:59.982733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:59.983118 (+ 385us) server_negotiation.cc:407] Connection header received
0928 16:43:59.983285 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:59.983288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:59.983336 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:59.983422 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:59.984314 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:59.984318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:59.984320 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:59.984332 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:59.984369 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:59.984382 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:59.984476 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:59.984645 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 16:43:59.984705 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:43:59.987140 15037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc: No bootstrap required, opened a new log
I20260928 16:43:59.991472 15037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786d320bd4484487b59908b8058a3ccc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39233 } }
I20260928 16:43:59.991828 15037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:59.991884 15037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786d320bd4484487b59908b8058a3ccc, State: Initialized, Role: FOLLOWER
I20260928 16:43:59.992452 15037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786d320bd4484487b59908b8058a3ccc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39233 } }
I20260928 16:43:59.992657 15037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:59.992738 15037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:59.992877 15037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:59.994089 15037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786d320bd4484487b59908b8058a3ccc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39233 } }
I20260928 16:43:59.994513 15037 leader_election.cc:302] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 786d320bd4484487b59908b8058a3ccc; no voters:
I20260928 16:43:59.994952 15037 leader_election.cc:288] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:59.995229 15047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:59.995653 15047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc [term 1 LEADER]: Becoming Leader. State: Replica: 786d320bd4484487b59908b8058a3ccc, State: Running, Role: LEADER
I20260928 16:43:59.996078 15047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786d320bd4484487b59908b8058a3ccc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39233 } }
I20260928 16:43:59.996693 15037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:59.998107 15049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 786d320bd4484487b59908b8058a3ccc. Latest consensus state: current_term: 1 leader_uuid: "786d320bd4484487b59908b8058a3ccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786d320bd4484487b59908b8058a3ccc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39233 } } }
I20260928 16:43:59.998308 15049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:59.998831 15056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:59.998871 15048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "786d320bd4484487b59908b8058a3ccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786d320bd4484487b59908b8058a3ccc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39233 } } }
I20260928 16:44:00.000026 15048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:00.002956 15056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:00.010772 15056 catalog_manager.cc:1347] Generated new cluster ID: 03da295e179c4762886e08ae9026a333
I20260928 16:44:00.010888 15056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:00.038015 15056 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:00.038451 15056 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:00.047394 15056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 786d320bd4484487b59908b8058a3ccc: Generated new TSK 0
I20260928 16:44:00.048317 15056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:44:00.056867 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 14969
2026-09-28T16:44:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:00 dist-test-slave-nqgh krb5kdc[15068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:00 dist-test-slave-nqgh krb5kdc[15068](info): set up 2 sockets
Sep 28 16:44:00 dist-test-slave-nqgh krb5kdc[15068](info): commencing operation
krb5kdc: starting...
W20260928 16:44:01.040608 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:44:01 dist-test-slave-nqgh krb5kdc[15068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:44:01.179314 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:34491
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:46541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:34491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:01.316279 15081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:01.316593 15081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:01.316639 15081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:01.321185 15081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:01.321293 15081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:01.321314 15081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:01.321329 15081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:01.321345 15081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:01.327528 15081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:46541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:34491
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:34491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:01.328722 15081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:01.329784 15081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:01.339823 15089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:01.339859 15087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:01.339854 15086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:01.340298 15081 server_base.cc:1034] running on GCE node
I20260928 16:44:01.341092 15081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:01.342689 15081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:01.343927 15081 hybrid_clock.cc:648] HybridClock initialized: now 1790613841343901 us; error 46 us; skew 500 ppm
Sep 28 16:44:01 dist-test-slave-nqgh krb5kdc[15068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613841, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:01.347942 15081 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:44:01.349566 15081 webserver.cc:460] Webserver started at http://127.25.71.126:38185/ using document root <none> and password file <none>
I20260928 16:44:01.350356 15081 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:01.350440 15081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:01.350773 15081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:01.353646 15081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "5620fc8b4ee94957a41719f641be0387"
format_stamp: "Formatted at 2026-09-28 16:44:01 on dist-test-slave-nqgh"
server_key: "2377adcd63bc346fe79ac2f6728867d5"
server_key_iv: "c8d7b05a61ab84da797324050f972e52"
server_key_version: "encryptionkey@0"
I20260928 16:44:01.354395 15081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "5620fc8b4ee94957a41719f641be0387"
format_stamp: "Formatted at 2026-09-28 16:44:01 on dist-test-slave-nqgh"
server_key: "2377adcd63bc346fe79ac2f6728867d5"
server_key_iv: "c8d7b05a61ab84da797324050f972e52"
server_key_version: "encryptionkey@0"
I20260928 16:44:01.359514 15081 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 16:44:01.363837 15096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:01.365974 15081 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:44:01.366441 15081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "5620fc8b4ee94957a41719f641be0387"
format_stamp: "Formatted at 2026-09-28 16:44:01 on dist-test-slave-nqgh"
server_key: "2377adcd63bc346fe79ac2f6728867d5"
server_key_iv: "c8d7b05a61ab84da797324050f972e52"
server_key_version: "encryptionkey@0"
I20260928 16:44:01.366709 15081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:01.401891 15081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:01.406522 15081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:01.406802 15081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:01.416213 15081 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:34491
I20260928 16:44:01.416241 15148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:34491 every 8 connection(s)
I20260928 16:44:01.417759 15081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:01.418292 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 15081
I20260928 16:44:01.418444 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:44:01.418799 25885 external_mini_cluster.cc:1428] Setting key 095d87e749961e45cdb0e8dc58a24dff
I20260928 16:44:01.422211 15149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 16:44:01 dist-test-slave-nqgh krb5kdc[15068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:44:01.431005 15149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387: Bootstrap starting.
I20260928 16:44:01.434602 15149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:01.435913 15149 log.cc:824] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:01.436764 15152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:01.420655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50465 (local address 127.25.71.126:34491)
0928 16:44:01.421179 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:01.421200 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:01.421246 (+ 46us) server_negotiation.cc:407] Connection header received
0928 16:44:01.422030 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:01.422049 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:01.422500 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:01.422911 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:01.423790 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:01.424898 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:01.425780 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:01.426315 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:01.426485 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:01.429909 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:01.429916 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:01.429935 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:01.429979 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:01.432245 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:01.433266 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:01.433273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:01.433280 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:01.433488 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:01.434077 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:01.434081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:01.434082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:01.434694 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:01.435009 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:01.435453 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 16:44:01.435737 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":174,"threads_started":1}
I20260928 16:44:01.439246 15149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387: No bootstrap required, opened a new log
I20260928 16:44:01.441414 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:44:01.444519 15149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5620fc8b4ee94957a41719f641be0387" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 34491 } }
I20260928 16:44:01.444932 15149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:01.445014 15149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5620fc8b4ee94957a41719f641be0387, State: Initialized, Role: FOLLOWER
I20260928 16:44:01.445628 15149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5620fc8b4ee94957a41719f641be0387" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 34491 } }
I20260928 16:44:01.445838 15149 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:01.445923 15149 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:01.446058 15149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:01.447616 15149 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5620fc8b4ee94957a41719f641be0387" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 34491 } }
I20260928 16:44:01.448199 15149 leader_election.cc:302] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5620fc8b4ee94957a41719f641be0387; no voters:
I20260928 16:44:01.448735 15149 leader_election.cc:288] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:01.449024 15159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:01.449563 15159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387 [term 1 LEADER]: Becoming Leader. State: Replica: 5620fc8b4ee94957a41719f641be0387, State: Running, Role: LEADER
I20260928 16:44:01.450073 15159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5620fc8b4ee94957a41719f641be0387" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 34491 } }
I20260928 16:44:01.450664 15149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:01.452090 15160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5620fc8b4ee94957a41719f641be0387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5620fc8b4ee94957a41719f641be0387" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 34491 } } }
I20260928 16:44:01.452270 15160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:01.452602 15161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5620fc8b4ee94957a41719f641be0387. Latest consensus state: current_term: 1 leader_uuid: "5620fc8b4ee94957a41719f641be0387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5620fc8b4ee94957a41719f641be0387" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 34491 } } }
I20260928 16:44:01.452728 15161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:01.453030 15168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:01.455488 15152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:01.443874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37567 (local address 127.25.71.126:34491)
0928 16:44:01.444032 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:01.444037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:01.444115 (+ 78us) server_negotiation.cc:407] Connection header received
0928 16:44:01.444251 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:01.444254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:01.444324 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:01.444452 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:01.445538 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:01.446385 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:01.447490 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:01.447873 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:01.448006 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:01.449997 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:01.450001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:01.450004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:01.450048 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:01.452056 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:01.453036 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:01.453040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:01.453041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:01.453149 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:01.453748 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:01.453754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:01.453756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:01.454045 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:01.454211 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:01.455190 (+ 979us) server_negotiation.cc:299] Negotiation successful
0928 16:44:01.455313 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:44:01.457059 15168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:01.465396 15168 catalog_manager.cc:1347] Generated new cluster ID: d05982ed4c444b9f9f4b2bd0c30d422f
I20260928 16:44:01.465502 15168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:01.493731 15168 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:01.495469 15168 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:01.501931 15168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5620fc8b4ee94957a41719f641be0387: Generated new TSK 0
I20260928 16:44:01.502969 15168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:44:01.515146 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 15081
W20260928 16:44:01.521085 15066 connection.cc:535] client connection to 127.25.71.126:34491 recv error: Network error: failed to read from TLS socket (remote: 127.25.71.126:34491): Connection reset by peer (error 104)
2026-09-28T16:44:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1456 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:01 dist-test-slave-nqgh krb5kdc[15180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:01 dist-test-slave-nqgh krb5kdc[15180](info): set up 2 sockets
Sep 28 16:44:01 dist-test-slave-nqgh krb5kdc[15180](info): commencing operation
krb5kdc: starting...
W20260928 16:44:02.504776 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:44:02 dist-test-slave-nqgh krb5kdc[15180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:44:02.642773 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:37733
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:44727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:37733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:02.779067 15193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:02.779459 15193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:02.779531 15193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:02.784164 15193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:02.784276 15193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:02.784304 15193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:02.784328 15193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:02.784361 15193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:02.791100 15193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:44727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:37733
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:37733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:02.792634 15193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:02.793988 15193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:02.805066 15199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:02.805078 15201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:02.805066 15198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:02.805151 15193 server_base.cc:1034] running on GCE node
I20260928 16:44:02.806398 15193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:02.808022 15193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:02.809242 15193 hybrid_clock.cc:648] HybridClock initialized: now 1790613842809224 us; error 53 us; skew 500 ppm
Sep 28 16:44:02 dist-test-slave-nqgh krb5kdc[15180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613842, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:02.812881 15193 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:44:02.814056 15193 webserver.cc:460] Webserver started at http://127.25.71.126:34369/ using document root <none> and password file <none>
I20260928 16:44:02.814879 15193 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:02.814970 15193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:02.815193 15193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:02.817682 15193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "08ba3a9ed3f547d6a7a5a9ee1b61ce36"
format_stamp: "Formatted at 2026-09-28 16:44:02 on dist-test-slave-nqgh"
server_key: "e8160eaf29a76b9608541d8c2b03124f"
server_key_iv: "e2b4edc0754be750109e84dde8f9fb1f"
server_key_version: "encryptionkey@0"
I20260928 16:44:02.818342 15193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "08ba3a9ed3f547d6a7a5a9ee1b61ce36"
format_stamp: "Formatted at 2026-09-28 16:44:02 on dist-test-slave-nqgh"
server_key: "e8160eaf29a76b9608541d8c2b03124f"
server_key_iv: "e2b4edc0754be750109e84dde8f9fb1f"
server_key_version: "encryptionkey@0"
I20260928 16:44:02.823328 15193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 16:44:02.826869 15208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:02.828761 15193 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:44:02.829044 15193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "08ba3a9ed3f547d6a7a5a9ee1b61ce36"
format_stamp: "Formatted at 2026-09-28 16:44:02 on dist-test-slave-nqgh"
server_key: "e8160eaf29a76b9608541d8c2b03124f"
server_key_iv: "e2b4edc0754be750109e84dde8f9fb1f"
server_key_version: "encryptionkey@0"
I20260928 16:44:02.829190 15193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:02.841948 15193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:02.845611 15193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:02.845829 15193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:02.854266 15193 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:37733
I20260928 16:44:02.854373 15260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:37733 every 8 connection(s)
I20260928 16:44:02.855451 15193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:02.859624 15261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:02.861760 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 15193
I20260928 16:44:02.861915 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:44:02.862229 25885 external_mini_cluster.cc:1428] Setting key c23c2485038d41bc227e37a601293865
I20260928 16:44:02.868445 15261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36: Bootstrap starting.
I20260928 16:44:02.871909 15261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:02.872996 15261 log.cc:824] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:02.875206 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:02.864150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44475 (local address 127.25.71.126:37733)
0928 16:44:02.864815 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:02.864841 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:02.864902 (+ 61us) server_negotiation.cc:407] Connection header received
0928 16:44:02.865979 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:02.866011 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:02.866618 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:02.867173 (+ 555us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:02.867688 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:02.869251 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:02.870475 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:02.871281 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:02.871471 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:02.872603 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:02.872611 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:02.872633 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:02.873245 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:02.873336 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:02.873373 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:02.873538 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:02.873838 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 16:44:02.874014 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":187,"threads_started":1}
I20260928 16:44:02.876082 15261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36: No bootstrap required, opened a new log
I20260928 16:44:02.879302 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:44:02.880753 15261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ba3a9ed3f547d6a7a5a9ee1b61ce36" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37733 } }
I20260928 16:44:02.881098 15261 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:02.881165 15261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ba3a9ed3f547d6a7a5a9ee1b61ce36, State: Initialized, Role: FOLLOWER
I20260928 16:44:02.881752 15261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ba3a9ed3f547d6a7a5a9ee1b61ce36" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37733 } }
I20260928 16:44:02.881910 15261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:02.881986 15261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:02.882104 15261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:02.883679 15261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ba3a9ed3f547d6a7a5a9ee1b61ce36" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37733 } }
I20260928 16:44:02.884135 15261 leader_election.cc:302] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08ba3a9ed3f547d6a7a5a9ee1b61ce36; no voters:
I20260928 16:44:02.884539 15261 leader_election.cc:288] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:02.884790 15271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:02.885202 15271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36 [term 1 LEADER]: Becoming Leader. State: Replica: 08ba3a9ed3f547d6a7a5a9ee1b61ce36, State: Running, Role: LEADER
I20260928 16:44:02.885746 15271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ba3a9ed3f547d6a7a5a9ee1b61ce36" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37733 } }
I20260928 16:44:02.886466 15261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:02.886951 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:02.881555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54923 (local address 127.25.71.126:37733)
0928 16:44:02.881761 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:02.881766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:02.881995 (+ 229us) server_negotiation.cc:407] Connection header received
0928 16:44:02.882218 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:02.882223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:02.882299 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:02.882415 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:02.882722 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:02.883541 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:02.884622 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:02.885055 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:02.885164 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:02.886341 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:02.886345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:02.886347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:02.886366 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:02.886403 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:02.886416 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:02.886518 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:02.886739 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 16:44:02.886797 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 16:44:02.888087 15272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08ba3a9ed3f547d6a7a5a9ee1b61ce36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ba3a9ed3f547d6a7a5a9ee1b61ce36" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37733 } } }
I20260928 16:44:02.888306 15272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:02.888087 15273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08ba3a9ed3f547d6a7a5a9ee1b61ce36. Latest consensus state: current_term: 1 leader_uuid: "08ba3a9ed3f547d6a7a5a9ee1b61ce36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ba3a9ed3f547d6a7a5a9ee1b61ce36" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 37733 } } }
I20260928 16:44:02.888370 15273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:02.888885 15282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:02.892441 15282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:02.898486 15282 catalog_manager.cc:1347] Generated new cluster ID: 0980ad01e5b84a13ba42168ca8284d4b
I20260928 16:44:02.898590 15282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:02.922290 15282 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:02.923663 15282 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:02.932467 15282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08ba3a9ed3f547d6a7a5a9ee1b61ce36: Generated new TSK 0
I20260928 16:44:02.933338 15282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:44:02.946236 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 15193
W20260928 16:44:02.952023 15178 connection.cc:535] client connection to 127.25.71.126:37733 recv error: Network error: failed to read from TLS socket (remote: 127.25.71.126:37733): Connection reset by peer (error 104)
2026-09-28T16:44:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:03 dist-test-slave-nqgh krb5kdc[15292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:03 dist-test-slave-nqgh krb5kdc[15292](info): set up 2 sockets
Sep 28 16:44:03 dist-test-slave-nqgh krb5kdc[15292](info): commencing operation
krb5kdc: starting...
W20260928 16:44:03.924702 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:44:04 dist-test-slave-nqgh krb5kdc[15292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:44:04.067710 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:34119
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:34119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:04.205639 15305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:04.206018 15305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:04.206089 15305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:04.210693 15305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:04.210822 15305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:04.210851 15305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:04.210875 15305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:04.210904 15305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:04.217835 15305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:34119
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:34119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:04.219537 15305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:04.221026 15305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:04.232214 15313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:04.232273 15310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:04.232273 15311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:04.232403 15305 server_base.cc:1034] running on GCE node
I20260928 16:44:04.233505 15305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:04.235170 15305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:04.236379 15305 hybrid_clock.cc:648] HybridClock initialized: now 1790613844236355 us; error 45 us; skew 500 ppm
Sep 28 16:44:04 dist-test-slave-nqgh krb5kdc[15292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613844, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:04.244344 15305 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:44:04.245690 15305 webserver.cc:460] Webserver started at http://127.25.71.126:42019/ using document root <none> and password file <none>
I20260928 16:44:04.246433 15305 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:04.246511 15305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:04.246804 15305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:04.249560 15305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "886db77d069e48a68ea58d0df7cd0a4b"
format_stamp: "Formatted at 2026-09-28 16:44:04 on dist-test-slave-nqgh"
server_key: "4563982d499c102a57f78f1823887e74"
server_key_iv: "9811bdf46ae60f4a55ccca4900bad432"
server_key_version: "encryptionkey@0"
I20260928 16:44:04.250212 15305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "886db77d069e48a68ea58d0df7cd0a4b"
format_stamp: "Formatted at 2026-09-28 16:44:04 on dist-test-slave-nqgh"
server_key: "4563982d499c102a57f78f1823887e74"
server_key_iv: "9811bdf46ae60f4a55ccca4900bad432"
server_key_version: "encryptionkey@0"
I20260928 16:44:04.255509 15305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 16:44:04.259362 15320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:04.261399 15305 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:44:04.261633 15305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "886db77d069e48a68ea58d0df7cd0a4b"
format_stamp: "Formatted at 2026-09-28 16:44:04 on dist-test-slave-nqgh"
server_key: "4563982d499c102a57f78f1823887e74"
server_key_iv: "9811bdf46ae60f4a55ccca4900bad432"
server_key_version: "encryptionkey@0"
I20260928 16:44:04.261803 15305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:04.268231 15305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:04.272122 15305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:04.272356 15305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:04.282378 15305 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:34119
I20260928 16:44:04.282423 15372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:34119 every 8 connection(s)
I20260928 16:44:04.283807 15305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:04.286974 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 15305
I20260928 16:44:04.287233 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:44:04.287292 15373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:04.287612 25885 external_mini_cluster.cc:1428] Setting key 6f49b20763b63a007ddda53209a2545e
I20260928 16:44:04.294739 15376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:04.289435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45093 (local address 127.25.71.126:34119)
0928 16:44:04.290012 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:04.290031 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:04.290081 (+ 50us) server_negotiation.cc:407] Connection header received
0928 16:44:04.291076 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:04.291103 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:04.291521 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:04.291937 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:04.293027 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:04.293034 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:04.293049 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:04.293063 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:04.293135 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:04.293160 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:04.293309 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:04.293430 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 16:44:04.293636 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":173,"threads_started":1}
I20260928 16:44:04.295650 15373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b: Bootstrap starting.
I20260928 16:44:04.298270 15373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:04.298826 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:44:04.299160 15373 log.cc:824] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:04.301844 15373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b: No bootstrap required, opened a new log
I20260928 16:44:04.303721 15376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:04.301488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34145 (local address 127.25.71.126:34119)
0928 16:44:04.301641 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:04.301645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:04.301821 (+ 176us) server_negotiation.cc:407] Connection header received
0928 16:44:04.302001 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:04.302005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:04.302047 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:04.302129 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:04.303169 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:04.303172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:04.303174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:04.303190 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:04.303227 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:04.303241 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:04.303338 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:04.303480 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 16:44:04.303523 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:44:04.306260 15373 raft_consensus.cc:348] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886db77d069e48a68ea58d0df7cd0a4b" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 34119 } }
I20260928 16:44:04.306522 15373 raft_consensus.cc:374] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:04.306564 15373 raft_consensus.cc:729] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886db77d069e48a68ea58d0df7cd0a4b, State: Initialized, Role: FOLLOWER
I20260928 16:44:04.307053 15373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886db77d069e48a68ea58d0df7cd0a4b" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 34119 } }
I20260928 16:44:04.307196 15373 raft_consensus.cc:388] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:04.307266 15373 raft_consensus.cc:482] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:04.307375 15373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:04.308370 15373 raft_consensus.cc:504] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886db77d069e48a68ea58d0df7cd0a4b" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 34119 } }
I20260928 16:44:04.308725 15373 leader_election.cc:302] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 886db77d069e48a68ea58d0df7cd0a4b; no voters:
I20260928 16:44:04.309063 15373 leader_election.cc:288] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:04.309408 15383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:04.309739 15383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b [term 1 LEADER]: Becoming Leader. State: Replica: 886db77d069e48a68ea58d0df7cd0a4b, State: Running, Role: LEADER
I20260928 16:44:04.310184 15383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886db77d069e48a68ea58d0df7cd0a4b" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 34119 } }
I20260928 16:44:04.310384 15373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:04.312431 15384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "886db77d069e48a68ea58d0df7cd0a4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886db77d069e48a68ea58d0df7cd0a4b" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 34119 } } }
I20260928 16:44:04.312597 15384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:04.312914 15385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 886db77d069e48a68ea58d0df7cd0a4b. Latest consensus state: current_term: 1 leader_uuid: "886db77d069e48a68ea58d0df7cd0a4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886db77d069e48a68ea58d0df7cd0a4b" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 34119 } } }
I20260928 16:44:04.313717 15385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b [sys.catalog]: This master's current role is: LEADER
W20260928 16:44:04.317154 15397 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:44:04.317251 15397 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:44:04.317456 15398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:04.318615 15398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:04.326220 15398 catalog_manager.cc:1347] Generated new cluster ID: d0e68718bcdc4ed9977475cebdcb4768
I20260928 16:44:04.326333 15398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:04.335497 15398 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:04.335991 15398 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:04.345396 15398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 886db77d069e48a68ea58d0df7cd0a4b: Generated new TSK 0
I20260928 16:44:04.346470 15398 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:44:04.351630 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 15305
2026-09-28T16:44:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:04 dist-test-slave-nqgh krb5kdc[15404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:04 dist-test-slave-nqgh krb5kdc[15404](info): set up 2 sockets
Sep 28 16:44:04 dist-test-slave-nqgh krb5kdc[15404](info): commencing operation
krb5kdc: starting...
W20260928 16:44:05.332736 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:44:05 dist-test-slave-nqgh krb5kdc[15404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:44:05.470691 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:46051
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:33165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:46051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:05.608875 15417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:05.609270 15417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:05.609340 15417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:05.613857 15417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:05.613965 15417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:05.613993 15417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:05.614015 15417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:05.614037 15417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:05.620386 15417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:33165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:46051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:46051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:05.621940 15417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:05.623366 15417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:05.634263 15422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:05.634263 15425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:05.634334 15417 server_base.cc:1034] running on GCE node
W20260928 16:44:05.634272 15423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:05.635390 15417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:05.636802 15417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:05.638011 15417 hybrid_clock.cc:648] HybridClock initialized: now 1790613845638021 us; error 97 us; skew 500 ppm
Sep 28 16:44:05 dist-test-slave-nqgh krb5kdc[15404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613845, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:05.641291 15417 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:44:05.642577 15417 webserver.cc:460] Webserver started at http://127.25.71.126:33057/ using document root <none> and password file <none>
I20260928 16:44:05.643311 15417 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:05.643400 15417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:05.643666 15417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:05.646029 15417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "c95e73cda7604497babb44ca13de938a"
format_stamp: "Formatted at 2026-09-28 16:44:05 on dist-test-slave-nqgh"
server_key: "35d86cf63da320bd8ecb603d336b9cf9"
server_key_iv: "6c337fa091d7cbbb2c44aa9e1634b0de"
server_key_version: "encryptionkey@0"
I20260928 16:44:05.646780 15417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "c95e73cda7604497babb44ca13de938a"
format_stamp: "Formatted at 2026-09-28 16:44:05 on dist-test-slave-nqgh"
server_key: "35d86cf63da320bd8ecb603d336b9cf9"
server_key_iv: "6c337fa091d7cbbb2c44aa9e1634b0de"
server_key_version: "encryptionkey@0"
I20260928 16:44:05.651635 15417 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 16:44:05.655503 15432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:05.657371 15417 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:44:05.657565 15417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "c95e73cda7604497babb44ca13de938a"
format_stamp: "Formatted at 2026-09-28 16:44:05 on dist-test-slave-nqgh"
server_key: "35d86cf63da320bd8ecb603d336b9cf9"
server_key_iv: "6c337fa091d7cbbb2c44aa9e1634b0de"
server_key_version: "encryptionkey@0"
I20260928 16:44:05.657706 15417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:05.674865 15417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:05.678196 15417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:05.678411 15417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:05.686582 15417 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:46051
I20260928 16:44:05.687008 15484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:46051 every 8 connection(s)
I20260928 16:44:05.687798 15417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:05.690110 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 15417
I20260928 16:44:05.690255 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:44:05.690567 25885 external_mini_cluster.cc:1428] Setting key 1ff246dc17890a97a4e14a171941b6d3
I20260928 16:44:05.692147 15485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:05.701282 15485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a: Bootstrap starting.
Sep 28 16:44:05 dist-test-slave-nqgh krb5kdc[15404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:44:05.704869 15485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:05.705984 15485 log.cc:824] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:05.709113 15485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a: No bootstrap required, opened a new log
I20260928 16:44:05.709486 15488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:05.692380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59405 (local address 127.25.71.126:46051)
0928 16:44:05.692991 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:05.693016 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:05.693100 (+ 84us) server_negotiation.cc:407] Connection header received
0928 16:44:05.694144 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:05.694173 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:05.694757 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:05.695244 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:05.696113 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.697851 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.699125 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.699926 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.700144 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:05.703039 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:05.703049 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:05.703072 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:05.703123 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:05.705391 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:05.706080 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:05.706087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:05.706095 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:05.706260 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:05.706692 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:05.706696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:05.706698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:05.707317 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:05.707540 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:05.707853 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 16:44:05.708334 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":169,"threads_started":1}
I20260928 16:44:05.713538 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:44:05.714201 15485 raft_consensus.cc:348] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95e73cda7604497babb44ca13de938a" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 46051 } }
I20260928 16:44:05.714555 15485 raft_consensus.cc:374] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:05.714680 15485 raft_consensus.cc:729] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95e73cda7604497babb44ca13de938a, State: Initialized, Role: FOLLOWER
I20260928 16:44:05.715287 15485 consensus_queue.cc:260] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95e73cda7604497babb44ca13de938a" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 46051 } }
I20260928 16:44:05.715451 15485 raft_consensus.cc:388] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:05.715526 15485 raft_consensus.cc:482] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:05.715643 15485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:05.717149 15485 raft_consensus.cc:504] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95e73cda7604497babb44ca13de938a" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 46051 } }
I20260928 16:44:05.717679 15485 leader_election.cc:302] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c95e73cda7604497babb44ca13de938a; no voters:
I20260928 16:44:05.718161 15485 leader_election.cc:288] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:05.718380 15495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:05.718772 15495 raft_consensus.cc:686] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a [term 1 LEADER]: Becoming Leader. State: Replica: c95e73cda7604497babb44ca13de938a, State: Running, Role: LEADER
I20260928 16:44:05.719332 15495 consensus_queue.cc:237] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95e73cda7604497babb44ca13de938a" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 46051 } }
I20260928 16:44:05.720043 15485 sys_catalog.cc:564] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:05.721954 15496 sys_catalog.cc:455] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c95e73cda7604497babb44ca13de938a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95e73cda7604497babb44ca13de938a" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 46051 } } }
I20260928 16:44:05.722322 15497 sys_catalog.cc:455] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c95e73cda7604497babb44ca13de938a. Latest consensus state: current_term: 1 leader_uuid: "c95e73cda7604497babb44ca13de938a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95e73cda7604497babb44ca13de938a" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 46051 } } }
I20260928 16:44:05.722996 15497 sys_catalog.cc:458] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:05.722474 15496 sys_catalog.cc:458] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:05.724519 15507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:05.726400 15488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:05.716044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56439 (local address 127.25.71.126:46051)
0928 16:44:05.716226 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:05.716231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:05.716253 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:44:05.716404 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:05.716407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:05.716469 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:05.716581 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:05.717271 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.718088 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.719381 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.719784 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.719925 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:05.721902 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:05.721907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:05.721910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:05.721949 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:05.724598 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:05.725175 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:05.725179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:05.725181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:05.725269 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:05.725517 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:05.725521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:05.725523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:05.725736 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:05.725856 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:05.726101 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 16:44:05.726226 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 16:44:05.728292 15507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:05.735752 15507 catalog_manager.cc:1347] Generated new cluster ID: 9965c723fe5748738b4ef8ab72239d5a
I20260928 16:44:05.735865 15507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:05.757558 15507 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:05.758849 15507 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:05.775120 15507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c95e73cda7604497babb44ca13de938a: Generated new TSK 0
I20260928 16:44:05.776350 15507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:44:05.788384 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 15417
2026-09-28T16:44:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:05 dist-test-slave-nqgh krb5kdc[15516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:05 dist-test-slave-nqgh krb5kdc[15516](info): set up 2 sockets
Sep 28 16:44:05 dist-test-slave-nqgh krb5kdc[15516](info): commencing operation
krb5kdc: starting...
W20260928 16:44:06.760767 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:44:06 dist-test-slave-nqgh krb5kdc[15516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:44:06.898538 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:36149
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:36495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:36149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:07.035917 15529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:07.036352 15529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:07.036425 15529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:07.041181 15529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:07.041317 15529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:07.041348 15529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:07.041371 15529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:07.041393 15529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:07.047948 15529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:36495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:36149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:36149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:07.049511 15529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:07.050938 15529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:07.061890 15535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:07.062052 15534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:07.062346 15529 server_base.cc:1034] running on GCE node
W20260928 16:44:07.062479 15537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:07.063575 15529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:07.065179 15529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:07.068526 15529 hybrid_clock.cc:648] HybridClock initialized: now 1790613847068459 us; error 82 us; skew 500 ppm
Sep 28 16:44:07 dist-test-slave-nqgh krb5kdc[15516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613847, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:07.072716 15529 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:44:07.074110 15529 webserver.cc:460] Webserver started at http://127.25.71.126:35761/ using document root <none> and password file <none>
I20260928 16:44:07.074934 15529 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:07.075028 15529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:07.075307 15529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:07.078178 15529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "4c9e84794e554eeba64d69be1f04f5ee"
format_stamp: "Formatted at 2026-09-28 16:44:07 on dist-test-slave-nqgh"
server_key: "e0552bd30f266b7bf41e33fe754a8a14"
server_key_iv: "bdab054ad6cc67e631eb7961f9ae6ec5"
server_key_version: "encryptionkey@0"
I20260928 16:44:07.079012 15529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "4c9e84794e554eeba64d69be1f04f5ee"
format_stamp: "Formatted at 2026-09-28 16:44:07 on dist-test-slave-nqgh"
server_key: "e0552bd30f266b7bf41e33fe754a8a14"
server_key_iv: "bdab054ad6cc67e631eb7961f9ae6ec5"
server_key_version: "encryptionkey@0"
I20260928 16:44:07.084931 15529 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 16:44:07.088789 15544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:07.090826 15529 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:44:07.091044 15529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "4c9e84794e554eeba64d69be1f04f5ee"
format_stamp: "Formatted at 2026-09-28 16:44:07 on dist-test-slave-nqgh"
server_key: "e0552bd30f266b7bf41e33fe754a8a14"
server_key_iv: "bdab054ad6cc67e631eb7961f9ae6ec5"
server_key_version: "encryptionkey@0"
I20260928 16:44:07.091218 15529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:07.120146 15529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:07.124743 15529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:07.125010 15529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:07.134660 15529 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:36149
I20260928 16:44:07.134778 15596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:36149 every 8 connection(s)
I20260928 16:44:07.136029 15529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:07.138011 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 15529
I20260928 16:44:07.138278 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:44:07.138713 25885 external_mini_cluster.cc:1428] Setting key ca7f01f9250c4151de3419d45f60a03e
I20260928 16:44:07.140086 15597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:07.148132 15597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee: Bootstrap starting.
I20260928 16:44:07.151240 15597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:07.151889 15600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:07.140794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58549 (local address 127.25.71.126:36149)
0928 16:44:07.141504 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:07.141534 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:07.141594 (+ 60us) server_negotiation.cc:407] Connection header received
0928 16:44:07.142523 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:07.142549 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:07.143103 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:07.143562 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:07.144139 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:07.145683 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:07.146995 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:07.147755 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:07.147972 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:07.149266 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:07.149273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:07.149292 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:07.149904 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:07.149992 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:07.150026 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:07.150155 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:07.150486 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 16:44:07.150808 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":204,"threads_started":1}
I20260928 16:44:07.152215 15597 log.cc:824] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:07.155174 15597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee: No bootstrap required, opened a new log
I20260928 16:44:07.156451 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:44:07.159461 15597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9e84794e554eeba64d69be1f04f5ee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 36149 } }
I20260928 16:44:07.159785 15597 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:07.159843 15597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c9e84794e554eeba64d69be1f04f5ee, State: Initialized, Role: FOLLOWER
I20260928 16:44:07.160382 15597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9e84794e554eeba64d69be1f04f5ee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 36149 } }
I20260928 16:44:07.160565 15597 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:07.160645 15597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:07.160754 15597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:07.161964 15597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9e84794e554eeba64d69be1f04f5ee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 36149 } }
I20260928 16:44:07.162348 15597 leader_election.cc:302] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c9e84794e554eeba64d69be1f04f5ee; no voters:
I20260928 16:44:07.162752 15597 leader_election.cc:288] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:07.162959 15607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:07.163293 15607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee [term 1 LEADER]: Becoming Leader. State: Replica: 4c9e84794e554eeba64d69be1f04f5ee, State: Running, Role: LEADER
I20260928 16:44:07.163798 15607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9e84794e554eeba64d69be1f04f5ee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 36149 } }
I20260928 16:44:07.164353 15597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:07.164836 15600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:07.159376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48837 (local address 127.25.71.126:36149)
0928 16:44:07.159541 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:07.159547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:07.159670 (+ 123us) server_negotiation.cc:407] Connection header received
0928 16:44:07.159894 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:07.159900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:07.159979 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:07.160087 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:07.160476 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:07.161245 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:07.162392 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:07.162809 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:07.162932 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:07.164256 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:07.164261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:07.164264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:07.164283 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:07.164334 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:07.164349 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:07.164446 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:07.164576 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 16:44:07.164673 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:44:07.166231 15609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c9e84794e554eeba64d69be1f04f5ee. Latest consensus state: current_term: 1 leader_uuid: "4c9e84794e554eeba64d69be1f04f5ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9e84794e554eeba64d69be1f04f5ee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 36149 } } }
I20260928 16:44:07.166210 15608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c9e84794e554eeba64d69be1f04f5ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9e84794e554eeba64d69be1f04f5ee" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 36149 } } }
I20260928 16:44:07.166425 15608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:07.166425 15609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee [sys.catalog]: This master's current role is: LEADER
W20260928 16:44:07.170836 15622 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:44:07.171089 15622 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:44:07.171258 15621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:07.172267 15621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:07.179193 15621 catalog_manager.cc:1347] Generated new cluster ID: 035cef884a004ed1a369116d9eb9085b
I20260928 16:44:07.179280 15621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:07.203194 15621 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:07.204339 15621 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:07.234922 15621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c9e84794e554eeba64d69be1f04f5ee: Generated new TSK 0
I20260928 16:44:07.235823 15621 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:44:07.250242 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 15529
2026-09-28T16:44:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1460 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:07 dist-test-slave-nqgh krb5kdc[15628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:07 dist-test-slave-nqgh krb5kdc[15628](info): set up 2 sockets
Sep 28 16:44:07 dist-test-slave-nqgh krb5kdc[15628](info): commencing operation
krb5kdc: starting...
W20260928 16:44:08.224805 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:44:08 dist-test-slave-nqgh krb5kdc[15628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:44:08.366269 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:44027
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:44027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:08.502146 15641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:08.502596 15641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:08.502717 15641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:08.507463 15641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:08.507587 15641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:08.507619 15641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:08.507642 15641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:08.507663 15641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:08.514256 15641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:44027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:44027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:08.515883 15641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:08.517252 15641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:08.528453 15646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:08.528527 15641 server_base.cc:1034] running on GCE node
W20260928 16:44:08.528439 15649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:08.528439 15647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:08.529561 15641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:08.530956 15641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:08.532397 15641 hybrid_clock.cc:648] HybridClock initialized: now 1790613848532352 us; error 77 us; skew 500 ppm
Sep 28 16:44:08 dist-test-slave-nqgh krb5kdc[15628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613848, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:08.536108 15641 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:44:08.537269 15641 webserver.cc:460] Webserver started at http://127.25.71.126:45063/ using document root <none> and password file <none>
I20260928 16:44:08.537940 15641 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:08.538025 15641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:08.538249 15641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:08.540630 15641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "c5e85e2e3bda42cf85b23978234d4a17"
format_stamp: "Formatted at 2026-09-28 16:44:08 on dist-test-slave-nqgh"
server_key: "119552fa3a20b226aea36e01b1d4ac48"
server_key_iv: "b467d730c8710fc321c7f6534887d126"
server_key_version: "encryptionkey@0"
I20260928 16:44:08.541519 15641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "c5e85e2e3bda42cf85b23978234d4a17"
format_stamp: "Formatted at 2026-09-28 16:44:08 on dist-test-slave-nqgh"
server_key: "119552fa3a20b226aea36e01b1d4ac48"
server_key_iv: "b467d730c8710fc321c7f6534887d126"
server_key_version: "encryptionkey@0"
I20260928 16:44:08.546890 15641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 16:44:08.550203 15656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:08.552142 15641 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:44:08.552362 15641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "c5e85e2e3bda42cf85b23978234d4a17"
format_stamp: "Formatted at 2026-09-28 16:44:08 on dist-test-slave-nqgh"
server_key: "119552fa3a20b226aea36e01b1d4ac48"
server_key_iv: "b467d730c8710fc321c7f6534887d126"
server_key_version: "encryptionkey@0"
I20260928 16:44:08.552505 15641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:08.557894 15641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:08.561726 15641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:08.561954 15641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:08.570801 15641 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:44027
I20260928 16:44:08.570850 15708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:44027 every 8 connection(s)
I20260928 16:44:08.571965 15641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:08.574898 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 15641
I20260928 16:44:08.575054 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:44:08.575404 25885 external_mini_cluster.cc:1428] Setting key 3bbf78d0100a980c8489442b9bfe8662
I20260928 16:44:08.576251 15709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:08.583262 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.577279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37097 (local address 127.25.71.126:44027)
0928 16:44:08.577941 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:08.577968 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:08.578028 (+ 60us) server_negotiation.cc:407] Connection header received
0928 16:44:08.579123 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:08.579152 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:08.579648 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:08.580177 (+ 529us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:08.581212 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:08.581219 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:08.581238 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:08.581254 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:08.581346 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:08.581381 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:08.581592 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:08.581831 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 16:44:08.582033 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":200,"threads_started":1}
I20260928 16:44:08.585492 15709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17: Bootstrap starting.
I20260928 16:44:08.587327 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:44:08.588783 15709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:08.589804 15709 log.cc:824] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:08.591508 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.589524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59693 (local address 127.25.71.126:44027)
0928 16:44:08.589712 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:08.589718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:08.589800 (+ 82us) server_negotiation.cc:407] Connection header received
0928 16:44:08.589923 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:08.589926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:08.589986 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:08.590110 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:08.590967 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:08.590971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:08.590974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:08.590989 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:08.591031 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:08.591046 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:08.591251 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:08.591319 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 16:44:08.591368 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 16:44:08.592839 15709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17: No bootstrap required, opened a new log
I20260928 16:44:08.597327 15709 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e85e2e3bda42cf85b23978234d4a17" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44027 } }
I20260928 16:44:08.597702 15709 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:08.597743 15709 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e85e2e3bda42cf85b23978234d4a17, State: Initialized, Role: FOLLOWER
I20260928 16:44:08.598330 15709 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e85e2e3bda42cf85b23978234d4a17" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44027 } }
I20260928 16:44:08.598538 15709 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:08.598739 15709 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:08.598867 15709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:08.600242 15709 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e85e2e3bda42cf85b23978234d4a17" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44027 } }
I20260928 16:44:08.600684 15709 leader_election.cc:302] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5e85e2e3bda42cf85b23978234d4a17; no voters:
I20260928 16:44:08.601071 15709 leader_election.cc:288] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:08.601270 15719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:08.601598 15719 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17 [term 1 LEADER]: Becoming Leader. State: Replica: c5e85e2e3bda42cf85b23978234d4a17, State: Running, Role: LEADER
I20260928 16:44:08.602021 15719 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e85e2e3bda42cf85b23978234d4a17" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44027 } }
I20260928 16:44:08.602690 15709 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:08.604192 15721 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5e85e2e3bda42cf85b23978234d4a17. Latest consensus state: current_term: 1 leader_uuid: "c5e85e2e3bda42cf85b23978234d4a17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e85e2e3bda42cf85b23978234d4a17" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44027 } } }
I20260928 16:44:08.604372 15721 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:08.604749 15720 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5e85e2e3bda42cf85b23978234d4a17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e85e2e3bda42cf85b23978234d4a17" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 44027 } } }
I20260928 16:44:08.604858 15720 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:08.605059 15728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:08.608906 15728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:08.617233 15728 catalog_manager.cc:1347] Generated new cluster ID: ee8cb12c5c144c2a857fd07a508f7f33
I20260928 16:44:08.617369 15728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:08.627256 15728 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:08.627884 15728 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:08.639348 15728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5e85e2e3bda42cf85b23978234d4a17: Generated new TSK 0
I20260928 16:44:08.640369 15728 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:44:08.652411 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 15641
2026-09-28T16:44:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1403 ms)
[----------] 18 tests from AuthTokenIssuingTest (25705 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:08 dist-test-slave-nqgh krb5kdc[15740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:08 dist-test-slave-nqgh krb5kdc[15740](info): set up 2 sockets
Sep 28 16:44:08 dist-test-slave-nqgh krb5kdc[15740](info): commencing operation
krb5kdc: starting...
W20260928 16:44:09.632910 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:44:09 dist-test-slave-nqgh krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:44:09.775046 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:39553
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:39553,127.25.71.125:46495,127.25.71.124:38091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:09.896134 15753 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:44:09.896482 15753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:09.896521 15753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:09.896579 15753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:09.901182 15753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:09.901288 15753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:09.901319 15753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:09.901341 15753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:09.901360 15753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:09.907248 15753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:39553,127.25.71.125:46495,127.25.71.124:38091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:39553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:09.908500 15753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:09.909617 15753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:09.919613 15758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:09.919662 15759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:09.919822 15753 server_base.cc:1034] running on GCE node
W20260928 16:44:09.919646 15761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:09.920713 15753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:09.922405 15753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:09.923583 15753 hybrid_clock.cc:648] HybridClock initialized: now 1790613849923576 us; error 67 us; skew 500 ppm
Sep 28 16:44:09 dist-test-slave-nqgh krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613849, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:09.931200 15753 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:44:09.932643 15753 webserver.cc:460] Webserver started at http://127.25.71.126:40809/ using document root <none> and password file <none>
I20260928 16:44:09.933395 15753 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:09.933467 15753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:09.933691 15753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:09.936473 15753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "32160554974149ab9b0bc947acb589dd"
format_stamp: "Formatted at 2026-09-28 16:44:09 on dist-test-slave-nqgh"
server_key: "e407d991bf04239acef5002fc03b14e6"
server_key_iv: "3c532dca791717a978fa71f942355f02"
server_key_version: "encryptionkey@0"
I20260928 16:44:09.937147 15753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "32160554974149ab9b0bc947acb589dd"
format_stamp: "Formatted at 2026-09-28 16:44:09 on dist-test-slave-nqgh"
server_key: "e407d991bf04239acef5002fc03b14e6"
server_key_iv: "3c532dca791717a978fa71f942355f02"
server_key_version: "encryptionkey@0"
I20260928 16:44:09.942184 15753 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:44:09.945953 15768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:09.948007 15753 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:44:09.948216 15753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "32160554974149ab9b0bc947acb589dd"
format_stamp: "Formatted at 2026-09-28 16:44:09 on dist-test-slave-nqgh"
server_key: "e407d991bf04239acef5002fc03b14e6"
server_key_iv: "3c532dca791717a978fa71f942355f02"
server_key_version: "encryptionkey@0"
I20260928 16:44:09.948397 15753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:09.962028 15753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:09.966257 15753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:09.966513 15753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:09.976136 15753 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:39553
I20260928 16:44:09.976477 15820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:39553 every 8 connection(s)
I20260928 16:44:09.977526 15753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:09.980947 15821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:09.984277 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 15753
I20260928 16:44:09.984426 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:44:09.984747 25885 external_mini_cluster.cc:1428] Setting key ce2df3bb952e09b0e4df2a05ea113ecc
I20260928 16:44:09.985350 15821 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39553 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.125@KRBTEST.COM" created.
Sep 28 16:44:09 dist-test-slave-nqgh krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613849, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:44:10.006448 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:09.988369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:39553 (local address 127.0.0.1:58088)
0928 16:44:09.989069 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:09.989128 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:09.990286 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:09.991372 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:09.991378 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:09.991784 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:09.992123 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:09.992139 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:09.994091 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:09.994094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:09.995023 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:09.995035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:09.996122 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:09.996126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:09.996263 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:09.996975 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:09.996994 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:09.999718 (+ 2724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:10.002068 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:10.002085 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:10.002098 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:10.002458 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:10.002982 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:10.002985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:10.002986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:10.003225 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:10.004176 (+ 951us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:10.004184 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:10.004395 (+ 211us) client_negotiation.cc:769] Sending connection context
0928 16:44:10.004601 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 16:44:10.005031 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"spinlock_wait_cycles":35584,"thread_start_us":246,"threads_started":1}
I20260928 16:44:10.006444 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:09.988368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58088 (local address 127.25.71.126:39553)
0928 16:44:09.988919 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:09.988945 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:09.989286 (+ 341us) server_negotiation.cc:407] Connection header received
0928 16:44:09.990760 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:09.990768 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:09.991070 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:09.991301 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:09.992327 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:09.993881 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:09.995184 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:09.995902 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:09.996077 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:09.999886 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:09.999892 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:09.999904 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:09.999927 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:10.001877 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.002605 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.002613 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.002614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.002823 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.003353 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.003356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.003358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.003988 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:10.004165 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.004826 (+ 661us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.005196 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":375,"thread_start_us":153,"threads_started":1}
I20260928 16:44:10.009765 15821 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:44:10.010886 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.010463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:46495 (local address 127.0.0.1:42896)
0928 16:44:10.010689 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.010786 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:46495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 16:44:10.011210 15771 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.71.125:46495: connect: Connection refused (error 111)
I20260928 16:44:10.012389 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.012127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:46495 (local address 127.0.0.1:42910)
0928 16:44:10.012251 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.012308 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:46495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 16:44:10.012632 15821 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.125:46495: Network error: Client connection negotiation failed: client connection to 127.25.71.125:46495: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.125.keytab.
Entry for principal kudu/127.25.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.125.keytab.
I20260928 16:44:10.018378 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.125:46495
--webserver_interface=127.25.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.125.keytab
--principal=kudu/127.25.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:39553,127.25.71.125:46495,127.25.71.124:38091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:44:10.062022 15821 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } attempt: 1
I20260928 16:44:10.063308 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.062939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:46495 (local address 127.0.0.1:42926)
0928 16:44:10.063134 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.063205 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:46495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 16:44:10.064460 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.064185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:46495 (local address 127.0.0.1:42938)
0928 16:44:10.064324 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.064384 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:46495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 16:44:10.064622 15821 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.125:46495: Network error: Client connection negotiation failed: client connection to 127.25.71.125:46495: connect: Connection refused (error 111)
I20260928 16:44:10.132805 15821 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } attempt: 2
I20260928 16:44:10.134001 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.133583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:46495 (local address 127.0.0.1:42940)
0928 16:44:10.133771 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.133884 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:46495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 16:44:10.135162 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.134890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:46495 (local address 127.0.0.1:42950)
0928 16:44:10.135024 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.135087 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:46495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 16:44:10.135340 15821 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.125:46495: Network error: Client connection negotiation failed: client connection to 127.25.71.125:46495: connect: Connection refused (error 111)
W20260928 16:44:10.137418 15828 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:44:10.137768 15828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:10.137812 15828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:10.137887 15828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:10.142302 15828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:10.142400 15828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:10.142426 15828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:10.142447 15828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:10.142469 15828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:10.148423 15828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:39553,127.25.71.125:46495,127.25.71.124:38091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.125.keytab
--principal=kudu/127.25.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.125:46495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.71.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:10.149658 15828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:10.150795 15828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:10.160821 15833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:10.160830 15834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:10.160821 15836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:10.161125 15828 server_base.cc:1034] running on GCE node
I20260928 16:44:10.161983 15828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:10.163642 15828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:10.164883 15828 hybrid_clock.cc:648] HybridClock initialized: now 1790613850164847 us; error 60 us; skew 500 ppm
Sep 28 16:44:10 dist-test-slave-nqgh krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613850, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:10.169018 15828 init.cc:377] Logged in from keytab as kudu/127.25.71.125@KRBTEST.COM (short username kudu)
I20260928 16:44:10.170418 15828 webserver.cc:460] Webserver started at http://127.25.71.125:44845/ using document root <none> and password file <none>
I20260928 16:44:10.171252 15828 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:10.171342 15828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:10.171591 15828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:10.174398 15828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-1/data/instance:
uuid: "c7342e3be3cc4dc7a364181b07ab7c7f"
format_stamp: "Formatted at 2026-09-28 16:44:10 on dist-test-slave-nqgh"
server_key: "a871268f96b218290fed34f3ad154801"
server_key_iv: "97e50d570977ccfa8b8f0abd2a0c5e96"
server_key_version: "encryptionkey@0"
I20260928 16:44:10.175170 15828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-1/wal/instance:
uuid: "c7342e3be3cc4dc7a364181b07ab7c7f"
format_stamp: "Formatted at 2026-09-28 16:44:10 on dist-test-slave-nqgh"
server_key: "a871268f96b218290fed34f3ad154801"
server_key_iv: "97e50d570977ccfa8b8f0abd2a0c5e96"
server_key_version: "encryptionkey@0"
I20260928 16:44:10.180421 15828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 16:44:10.183961 15843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:10.185847 15828 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:44:10.186077 15828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-1/wal
uuid: "c7342e3be3cc4dc7a364181b07ab7c7f"
format_stamp: "Formatted at 2026-09-28 16:44:10 on dist-test-slave-nqgh"
server_key: "a871268f96b218290fed34f3ad154801"
server_key_iv: "97e50d570977ccfa8b8f0abd2a0c5e96"
server_key_version: "encryptionkey@0"
I20260928 16:44:10.186241 15828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:10.225490 15828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:10.226539 15821 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } attempt: 3
I20260928 16:44:10.227602 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.227236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:46495 (local address 127.0.0.1:42954)
0928 16:44:10.227414 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.227487 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:46495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 16:44:10.228755 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.228373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:46495 (local address 127.0.0.1:42966)
0928 16:44:10.228557 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.228620 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:46495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260928 16:44:10.229025 15821 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.125:46495: Network error: Client connection negotiation failed: client connection to 127.25.71.125:46495: connect: Connection refused (error 111)
I20260928 16:44:10.230543 15828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:10.230784 15828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:10.240163 15828 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.125:46495
I20260928 16:44:10.240489 15895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.125:46495 every 8 connection(s)
I20260928 16:44:10.241468 15828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-1/data/info.pb
I20260928 16:44:10.245191 15896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:10.248287 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 15828
I20260928 16:44:10.248452 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-1/wal/instance
I20260928 16:44:10.248840 25885 external_mini_cluster.cc:1428] Setting key 825b0ca5bc98320325c71ed9873f622b
I20260928 16:44:10.249485 15896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39553 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.71.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.124@KRBTEST.COM" created.
Sep 28 16:44:10 dist-test-slave-nqgh krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613850, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.125@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:44:10.269493 15899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.252610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:39553 (local address 127.0.0.1:58098)
0928 16:44:10.253473 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.253561 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:10.254922 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:10.255739 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:10.255750 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:10.256279 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:10.257071 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.257093 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.258196 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.258200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.259330 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.259344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.260101 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.260105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.260288 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.261129 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:10.261149 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:10.263186 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:10.265821 (+ 2635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:10.265830 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:10.265856 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:10.266312 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:10.266771 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:10.266774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:10.266776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:10.267023 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:10.267638 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:10.267645 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:10.268050 (+ 405us) client_negotiation.cc:769] Sending connection context
0928 16:44:10.268258 (+ 208us) client_negotiation.cc:240] Negotiation successful
0928 16:44:10.268609 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":174,"threads_started":1}
I20260928 16:44:10.270354 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.251956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58098 (local address 127.25.71.126:39553)
0928 16:44:10.252115 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.252119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.253722 (+ 1603us) server_negotiation.cc:407] Connection header received
0928 16:44:10.255289 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.255293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.255382 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.255467 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.257344 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.257989 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.259546 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.259875 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.260005 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.263532 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:10.263538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:10.263543 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:10.263600 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:10.265554 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.266478 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.266483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.266485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.266610 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.267154 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.267158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.267160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.267478 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:10.267605 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.269327 (+ 1722us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.269537 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:44:10.271893 15896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:44:10 dist-test-slave-nqgh krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613850, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.125@KRBTEST.COM for kudu/127.25.71.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.124.keytab.
Entry for principal kudu/127.25.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.124.keytab.
I20260928 16:44:10.283317 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.124:38091
--webserver_interface=127.25.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.124.keytab
--principal=kudu/127.25.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:39553,127.25.71.125:46495,127.25.71.124:38091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:44:10.284085 15899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.272462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:46495 (local address 127.0.0.1:42978)
0928 16:44:10.272680 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.272699 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:10.272828 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:10.273665 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:10.273668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:10.273903 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:10.274057 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.274065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.275223 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.275226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.275974 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.275985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.276805 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.276808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.276951 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.277594 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:10.277608 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:10.279318 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:10.281990 (+ 2672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:10.281995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:10.281997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:10.282291 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:10.282712 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:10.282715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:10.282718 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:10.282839 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:10.283363 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:10.283366 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:10.283517 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 16:44:10.283684 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 16:44:10.283867 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 16:44:10.284584 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.272658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42978 (local address 127.25.71.125:46495)
0928 16:44:10.273188 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.273199 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.273222 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:44:10.273272 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.273277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.273462 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.273598 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.274196 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.275074 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.276152 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.276640 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.276756 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.279583 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:10.279600 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:10.279621 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:10.279657 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:10.281810 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.282447 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.282455 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.282456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.282579 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.282935 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.282937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.282938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.283244 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:10.283361 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.283840 (+ 479us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.284037 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":270,"threads_started":1}
I20260928 16:44:10.287351 15896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:44:10.288338 15899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.287959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:38091 (local address 127.0.0.1:33144)
0928 16:44:10.288138 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.288240 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 16:44:10.288566 15845 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.71.124:38091: connect: Connection refused (error 111)
I20260928 16:44:10.289582 15899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.289358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:38091 (local address 127.0.0.1:33152)
0928 16:44:10.289469 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.289516 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 16:44:10.289737 15896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.124:38091: Network error: Client connection negotiation failed: client connection to 127.25.71.124:38091: connect: Connection refused (error 111)
I20260928 16:44:10.339079 15896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } attempt: 1
I20260928 16:44:10.340332 15899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.339966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:38091 (local address 127.0.0.1:33168)
0928 16:44:10.340156 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.340227 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 16:44:10.341244 15899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.340929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:38091 (local address 127.0.0.1:33170)
0928 16:44:10.341042 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.341096 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 16:44:10.341383 15896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.124:38091: Network error: Client connection negotiation failed: client connection to 127.25.71.124:38091: connect: Connection refused (error 111)
I20260928 16:44:10.372252 15821 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } attempt: 4
Sep 28 16:44:10 dist-test-slave-nqgh krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613849, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for kudu/127.25.71.125@KRBTEST.COM
I20260928 16:44:10.383699 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.373151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42980 (local address 127.25.71.125:46495)
0928 16:44:10.373336 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.373352 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.373377 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:44:10.373508 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.373511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.373578 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.373708 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.374286 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.375596 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.376640 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.377014 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.377143 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.380191 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:10.380196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:10.380199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:10.380241 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:10.381795 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.382320 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.382323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.382324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.382406 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.382705 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.382708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.382710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.382959 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:10.383051 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.383347 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.383484 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 16:44:10.383690 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.373028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:46495 (local address 127.0.0.1:42980)
0928 16:44:10.373260 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.373279 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:10.373395 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:10.373726 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:10.373728 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:10.373972 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:10.374139 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.374147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.375783 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.375786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.376485 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.376498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.377238 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.377241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.377360 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.377995 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:10.378009 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:10.379932 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:10.381956 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:10.381961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:10.381963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:10.382215 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:10.382490 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:10.382494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:10.382496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:10.382580 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:10.383051 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:10.383055 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:10.383178 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 16:44:10.383280 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 16:44:10.383479 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260928 16:44:10.385195 15821 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:44:10.386358 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.385966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:38091 (local address 127.0.0.1:33184)
0928 16:44:10.386158 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.386246 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 16:44:10.387250 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.387012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:38091 (local address 127.0.0.1:33200)
0928 16:44:10.387121 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.387182 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 16:44:10.387507 15821 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.124:38091: Network error: Client connection negotiation failed: client connection to 127.25.71.124:38091: connect: Connection refused (error 111)
W20260928 16:44:10.403905 15902 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:44:10.404247 15902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:10.404271 15902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:10.404323 15902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:10.409160 15902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:10.409261 15902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:10.409281 15902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:10.409307 15902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:10.409323 15902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:10.409585 15896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } attempt: 2
I20260928 16:44:10.410774 15899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.410369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:38091 (local address 127.0.0.1:33210)
0928 16:44:10.410553 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.410665 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 16:44:10.411926 15899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.411657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:38091 (local address 127.0.0.1:33224)
0928 16:44:10.411793 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.411852 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 16:44:10.412174 15896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.124:38091: Network error: Client connection negotiation failed: client connection to 127.25.71.124:38091: connect: Connection refused (error 111)
I20260928 16:44:10.415921 15902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:39553,127.25.71.125:46495,127.25.71.124:38091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.124.keytab
--principal=kudu/127.25.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.124:38091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.71.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:10.417138 15902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:10.418316 15902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:10.428416 15911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:10.428541 15902 server_base.cc:1034] running on GCE node
W20260928 16:44:10.428404 15908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:10.428395 15909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:10.429569 15902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:10.431331 15902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:10.432564 15902 hybrid_clock.cc:648] HybridClock initialized: now 1790613850432525 us; error 60 us; skew 500 ppm
Sep 28 16:44:10 dist-test-slave-nqgh krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613850, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:10.436256 15902 init.cc:377] Logged in from keytab as kudu/127.25.71.124@KRBTEST.COM (short username kudu)
I20260928 16:44:10.437610 15902 webserver.cc:460] Webserver started at http://127.25.71.124:40359/ using document root <none> and password file <none>
I20260928 16:44:10.438387 15902 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:10.438481 15902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:10.438824 15902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:10.441742 15902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-2/data/instance:
uuid: "1167282fac594599842835302ef0d373"
format_stamp: "Formatted at 2026-09-28 16:44:10 on dist-test-slave-nqgh"
server_key: "f0fc264a4f8b03d67540a26b2d635881"
server_key_iv: "b16c8a9f10369e5d9819be346a2f911f"
server_key_version: "encryptionkey@0"
I20260928 16:44:10.442497 15902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-2/wal/instance:
uuid: "1167282fac594599842835302ef0d373"
format_stamp: "Formatted at 2026-09-28 16:44:10 on dist-test-slave-nqgh"
server_key: "f0fc264a4f8b03d67540a26b2d635881"
server_key_iv: "b16c8a9f10369e5d9819be346a2f911f"
server_key_version: "encryptionkey@0"
I20260928 16:44:10.446728 15821 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } attempt: 1
I20260928 16:44:10.447896 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.447457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:38091 (local address 127.0.0.1:33232)
0928 16:44:10.447652 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.447738 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 16:44:10.448360 15902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:44:10.448768 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.448553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:38091 (local address 127.0.0.1:33238)
0928 16:44:10.448651 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.448709 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 16:44:10.449093 15821 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.124:38091: Network error: Client connection negotiation failed: client connection to 127.25.71.124:38091: connect: Connection refused (error 111)
I20260928 16:44:10.452018 15918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:10.454069 15902 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260928 16:44:10.454285 15902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-2/wal
uuid: "1167282fac594599842835302ef0d373"
format_stamp: "Formatted at 2026-09-28 16:44:10 on dist-test-slave-nqgh"
server_key: "f0fc264a4f8b03d67540a26b2d635881"
server_key_iv: "b16c8a9f10369e5d9819be346a2f911f"
server_key_version: "encryptionkey@0"
I20260928 16:44:10.454468 15902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:10.474095 15902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:10.478041 15902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:10.478295 15902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:10.487289 15902 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.124:38091
I20260928 16:44:10.487668 15970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.124:38091 every 8 connection(s)
I20260928 16:44:10.488528 15902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-2/data/info.pb
I20260928 16:44:10.491916 15971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:10.492540 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 15902
I20260928 16:44:10.492683 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613545667819-25885-0/minicluster-data/master-2/wal/instance
I20260928 16:44:10.493005 25885 external_mini_cluster.cc:1428] Setting key dad60c6065a129fc5f6a8841074972ab
I20260928 16:44:10.497102 15971 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39553 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:44:10.503360 15896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } attempt: 3
Sep 28 16:44:10 dist-test-slave-nqgh krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.124@KRBTEST.COM
Sep 28 16:44:10 dist-test-slave-nqgh krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613850, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.124@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
Sep 28 16:44:10 dist-test-slave-nqgh krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613850, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.125@KRBTEST.COM for kudu/127.25.71.124@KRBTEST.COM
I20260928 16:44:10.510520 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.499358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58104 (local address 127.25.71.126:39553)
0928 16:44:10.499538 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.499543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.499913 (+ 370us) server_negotiation.cc:407] Connection header received
0928 16:44:10.500160 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.500164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.500225 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.500323 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.501152 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.501731 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.502967 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.503205 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.503317 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.506730 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:10.506735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:10.506738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:10.506777 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:10.508502 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.509024 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.509028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.509030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.509121 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.509438 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.509442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.509443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.509680 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:10.509755 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.510221 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.510342 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:44:10.511351 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.499199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:39553 (local address 127.0.0.1:58104)
0928 16:44:10.499755 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.499821 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:10.500028 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:10.500391 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:10.500397 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:10.500827 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:10.501002 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.501009 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.501867 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.501871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.502843 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.502855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.503460 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.503465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.503603 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.504327 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:10.504349 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:10.506491 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:10.508629 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:10.508641 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:10.508649 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:10.508899 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:10.509234 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:10.509236 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:10.509238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:10.509329 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:10.509843 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:10.509851 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:10.509991 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 16:44:10.510125 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 16:44:10.510359 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":199,"threads_started":1}
I20260928 16:44:10.511370 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.495056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33254 (local address 127.25.71.124:38091)
0928 16:44:10.495675 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.495697 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.495751 (+ 54us) server_negotiation.cc:407] Connection header received
0928 16:44:10.496620 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.496646 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.497094 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.497499 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.498511 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.499772 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.500853 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.501411 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.501567 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.504700 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:10.504707 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:10.504725 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:10.504752 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:10.507043 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.507607 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.507617 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.507627 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.507825 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.508173 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.508176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.508178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.508860 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:10.509116 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.509514 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.509964 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":165,"threads_started":1}
I20260928 16:44:10.512939 15971 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:44:10.515174 15899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.503886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:38091 (local address 127.0.0.1:33262)
0928 16:44:10.504079 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.504092 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:10.504161 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:10.505021 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:10.505025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:10.505246 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:10.505376 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.505384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.506287 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.506291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.506979 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.506990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.507904 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.507908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.508006 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.508566 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:10.508580 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:10.510105 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:10.512878 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:10.512882 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:10.512884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:10.513129 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:10.513443 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:10.513446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:10.513448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:10.513536 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:10.514587 (+ 1051us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:10.514590 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:10.514745 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 16:44:10.514832 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 16:44:10.514988 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 16:44:10.515812 15976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.504290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33262 (local address 127.25.71.124:38091)
0928 16:44:10.504764 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.504773 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.504785 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:44:10.504818 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.504820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.504865 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.504950 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.505490 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.506113 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.507365 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.507778 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.507894 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.510268 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:10.510272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:10.510274 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:10.510297 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:10.512738 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.513252 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.513255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.513257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.513346 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.514185 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.514188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.514190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.514480 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:10.515387 (+ 907us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.515474 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.515615 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":380,"thread_start_us":167,"threads_started":1}
I20260928 16:44:10.516260 15821 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } attempt: 2
Sep 28 16:44:10 dist-test-slave-nqgh krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613850, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.124@KRBTEST.COM for kudu/127.25.71.125@KRBTEST.COM
I20260928 16:44:10.520938 15896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7342e3be3cc4dc7a364181b07ab7c7f: Bootstrap starting.
Sep 28 16:44:10 dist-test-slave-nqgh krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613849, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for kudu/127.25.71.124@KRBTEST.COM
Sep 28 16:44:10 dist-test-slave-nqgh krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:44:10.524623 15896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7342e3be3cc4dc7a364181b07ab7c7f: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:10.524794 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.513684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.25.71.125:46495)
0928 16:44:10.513852 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.513857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.513870 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:44:10.513905 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.513908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.513949 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.514030 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.514494 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.515311 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.516585 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.516949 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.517028 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.519513 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:10.519517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:10.519520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:10.519550 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:10.521227 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.523169 (+ 1942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.523175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.523177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.523303 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.523618 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.523621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.523623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.523953 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:10.524030 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.524409 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.524566 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":68}
I20260928 16:44:10.525620 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.513448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:46495 (local address 127.0.0.1:42984)
0928 16:44:10.513621 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.513633 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:10.513696 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:10.514057 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:10.514060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:10.514218 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:10.514366 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.514373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.515535 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.515538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.516442 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.516452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.517194 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.517197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.517291 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.517746 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:10.517761 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:10.519354 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:10.521353 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:10.521356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:10.521359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:10.523017 (+ 1658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:10.523422 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:10.523425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:10.523428 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:10.523519 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:10.524102 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:10.524105 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:10.524224 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 16:44:10.525248 (+ 1024us) client_negotiation.cc:240] Negotiation successful
0928 16:44:10.525445 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":2}
I20260928 16:44:10.525797 15896 log.cc:824] T 00000000000000000000000000000000 P c7342e3be3cc4dc7a364181b07ab7c7f: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:10.526398 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.516700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:38091 (local address 127.0.0.1:33264)
0928 16:44:10.516886 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.516903 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:10.516968 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:10.517282 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:10.517285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:10.517471 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:10.517580 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.517587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.518525 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.518528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.519338 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.519349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.520034 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.520037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.520120 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.520717 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:10.520731 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:10.522373 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:10.524754 (+ 2381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:10.524757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:10.524759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:10.525029 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:10.525324 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:10.525327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:10.525329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:10.525437 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:10.525908 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:10.525912 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:10.525989 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 16:44:10.526067 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 16:44:10.526222 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260928 16:44:10.527140 15976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.516959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33264 (local address 127.25.71.124:38091)
0928 16:44:10.517092 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.517097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.517108 (+ 11us) server_negotiation.cc:407] Connection header received
0928 16:44:10.517138 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.517140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.517181 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.517246 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.517706 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.518405 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.519579 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.519903 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.520375 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.522563 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:10.522568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:10.522571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:10.522600 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:10.524632 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.525124 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.525128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.525130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.525217 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.525548 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.525551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.525553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.525806 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:10.526731 (+ 925us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.526805 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.526983 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 16:44:10.527498 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.514649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58112 (local address 127.25.71.126:39553)
0928 16:44:10.515523 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.515528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.515543 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:44:10.515590 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.515593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.515644 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.515731 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.516550 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.517244 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.518773 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.519123 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.519218 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.523087 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:10.523090 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:10.523094 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:10.523124 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:10.524617 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.525421 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.525424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.525426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.525493 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.525770 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.525773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.525775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.526179 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:10.526423 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.527091 (+ 668us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.527346 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768}
I20260928 16:44:10.528220 15971 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:44:10.529564 15896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7342e3be3cc4dc7a364181b07ab7c7f: No bootstrap required, opened a new log
I20260928 16:44:10.531289 15821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd: Bootstrap starting.
Sep 28 16:44:10 dist-test-slave-nqgh krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613850, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.124@KRBTEST.COM for kudu/127.25.71.124@KRBTEST.COM
I20260928 16:44:10.534214 15896 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7342e3be3cc4dc7a364181b07ab7c7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32160554974149ab9b0bc947acb589dd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39553 } } peers { permanent_uuid: "c7342e3be3cc4dc7a364181b07ab7c7f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } } peers { permanent_uuid: "1167282fac594599842835302ef0d373" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } }
I20260928 16:44:10.534513 15896 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7342e3be3cc4dc7a364181b07ab7c7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7342e3be3cc4dc7a364181b07ab7c7f, State: Initialized, Role: FOLLOWER
I20260928 16:44:10.534946 15896 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7342e3be3cc4dc7a364181b07ab7c7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32160554974149ab9b0bc947acb589dd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39553 } } peers { permanent_uuid: "c7342e3be3cc4dc7a364181b07ab7c7f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } } peers { permanent_uuid: "1167282fac594599842835302ef0d373" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } }
I20260928 16:44:10.535144 15821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:10.536425 15821 log.cc:824] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:10.536669 15896 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7342e3be3cc4dc7a364181b07ab7c7f [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 16:44:10 dist-test-slave-nqgh krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.125@KRBTEST.COM
I20260928 16:44:10.538101 15976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.528950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33272 (local address 127.25.71.124:38091)
0928 16:44:10.529048 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.529052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.529179 (+ 127us) server_negotiation.cc:407] Connection header received
0928 16:44:10.529316 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.529319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.529356 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.529396 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.529947 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.530717 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.531788 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.532105 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.532179 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.534725 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:10.534729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:10.534731 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:10.534755 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:10.536371 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.536840 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.536843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.536845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.536920 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.537168 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.537171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.537172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.537378 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:10.537461 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.537799 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.537932 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 16:44:10.538652 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.528748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:38091 (local address 127.0.0.1:33272)
0928 16:44:10.529119 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.529133 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:10.529235 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:10.529537 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:10.529539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:10.529712 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:10.529804 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.529811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.530836 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.530838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.531648 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.531658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.532327 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.532331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.532415 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.532953 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:10.532968 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:10.534556 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:10.536503 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:10.536506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:10.536509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:10.536746 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:10.537007 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:10.537009 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:10.537011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:10.537088 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:10.537522 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:10.537524 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:10.537620 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 16:44:10.538328 (+ 708us) client_negotiation.cc:240] Negotiation successful
0928 16:44:10.538480 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325}
I20260928 16:44:10.542007 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.529116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42996 (local address 127.25.71.125:46495)
0928 16:44:10.529283 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.529288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.529459 (+ 171us) server_negotiation.cc:407] Connection header received
0928 16:44:10.529721 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.529724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.529776 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.529830 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.530819 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.531570 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.532734 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.533100 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.533178 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.537648 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:10.537652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:10.537656 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:10.537696 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:10.539605 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.540102 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.540106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.540108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.540222 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.540546 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.540549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.540551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.540805 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:10.540986 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.541694 (+ 708us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.541833 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 16:44:10.542107 15971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1167282fac594599842835302ef0d373: Bootstrap starting.
I20260928 16:44:10.545074 15971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1167282fac594599842835302ef0d373: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:10.542125 15828 master_runner.cc:186] Error getting master registration for 127.25.71.126:39553: OK, instance_id { permanent_uuid: "32160554974149ab9b0bc947acb589dd" instance_seqno: 1790613849973238 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:44:10.545755 15828 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:44:10.546039 15971 log.cc:824] T 00000000000000000000000000000000 P 1167282fac594599842835302ef0d373: Log is configured to *not* fsync() on all Append() calls
W20260928 16:44:10.546728 15990 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7342e3be3cc4dc7a364181b07ab7c7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:44:10.546831 15990 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:44:10.547113 15978 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7342e3be3cc4dc7a364181b07ab7c7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32160554974149ab9b0bc947acb589dd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39553 } } peers { permanent_uuid: "c7342e3be3cc4dc7a364181b07ab7c7f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } } peers { permanent_uuid: "1167282fac594599842835302ef0d373" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } } }
I20260928 16:44:10.547282 15978 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7342e3be3cc4dc7a364181b07ab7c7f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:10.549567 15821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd: No bootstrap required, opened a new log
I20260928 16:44:10.549722 15971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1167282fac594599842835302ef0d373: No bootstrap required, opened a new log
I20260928 16:44:10.550278 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:44:10.552006 15795 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "32160554974149ab9b0bc947acb589dd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58112
I20260928 16:44:10.554385 15821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32160554974149ab9b0bc947acb589dd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39553 } } peers { permanent_uuid: "c7342e3be3cc4dc7a364181b07ab7c7f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } } peers { permanent_uuid: "1167282fac594599842835302ef0d373" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } }
I20260928 16:44:10.554769 15821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32160554974149ab9b0bc947acb589dd, State: Initialized, Role: FOLLOWER
I20260928 16:44:10.554872 15971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1167282fac594599842835302ef0d373 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32160554974149ab9b0bc947acb589dd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39553 } } peers { permanent_uuid: "c7342e3be3cc4dc7a364181b07ab7c7f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } } peers { permanent_uuid: "1167282fac594599842835302ef0d373" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } }
I20260928 16:44:10.555228 15971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1167282fac594599842835302ef0d373 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1167282fac594599842835302ef0d373, State: Initialized, Role: FOLLOWER
I20260928 16:44:10.555167 15821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32160554974149ab9b0bc947acb589dd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39553 } } peers { permanent_uuid: "c7342e3be3cc4dc7a364181b07ab7c7f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } } peers { permanent_uuid: "1167282fac594599842835302ef0d373" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } }
I20260928 16:44:10.555626 15971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1167282fac594599842835302ef0d373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32160554974149ab9b0bc947acb589dd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39553 } } peers { permanent_uuid: "c7342e3be3cc4dc7a364181b07ab7c7f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } } peers { permanent_uuid: "1167282fac594599842835302ef0d373" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } }
I20260928 16:44:10.556164 15993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32160554974149ab9b0bc947acb589dd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39553 } } peers { permanent_uuid: "c7342e3be3cc4dc7a364181b07ab7c7f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } } peers { permanent_uuid: "1167282fac594599842835302ef0d373" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } } }
I20260928 16:44:10.556505 15993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:10.556967 15821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:10.557226 15795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 16:44:10.557400 15795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:10.557407 15971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1167282fac594599842835302ef0d373 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:10.559340 15795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32160554974149ab9b0bc947acb589dd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39553 } } peers { permanent_uuid: "c7342e3be3cc4dc7a364181b07ab7c7f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } } peers { permanent_uuid: "1167282fac594599842835302ef0d373" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } }
I20260928 16:44:10.561537 15870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32160554974149ab9b0bc947acb589dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c7342e3be3cc4dc7a364181b07ab7c7f"
I20260928 16:44:10.561856 15870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7342e3be3cc4dc7a364181b07ab7c7f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:10.562873 15795 leader_election.cc:288] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [CANDIDATE]: Term 1 election: Requested vote from peers c7342e3be3cc4dc7a364181b07ab7c7f (127.25.71.125:46495), 1167282fac594599842835302ef0d373 (127.25.71.124:38091)
I20260928 16:44:10.563942 15870 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c7342e3be3cc4dc7a364181b07ab7c7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32160554974149ab9b0bc947acb589dd in term 1.
I20260928 16:44:10.565702 15771 leader_election.cc:302] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32160554974149ab9b0bc947acb589dd, c7342e3be3cc4dc7a364181b07ab7c7f; no voters:
I20260928 16:44:10.561534 15945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32160554974149ab9b0bc947acb589dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1167282fac594599842835302ef0d373"
I20260928 16:44:10.566272 15945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1167282fac594599842835302ef0d373 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:10.567209 15993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:10.567534 15993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [term 1 LEADER]: Becoming Leader. State: Replica: 32160554974149ab9b0bc947acb589dd, State: Running, Role: LEADER
I20260928 16:44:10.567957 15993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32160554974149ab9b0bc947acb589dd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39553 } } peers { permanent_uuid: "c7342e3be3cc4dc7a364181b07ab7c7f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } } peers { permanent_uuid: "1167282fac594599842835302ef0d373" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } }
I20260928 16:44:10.568290 15945 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1167282fac594599842835302ef0d373 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32160554974149ab9b0bc947acb589dd in term 1.
I20260928 16:44:10.569686 15994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1167282fac594599842835302ef0d373 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32160554974149ab9b0bc947acb589dd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39553 } } peers { permanent_uuid: "c7342e3be3cc4dc7a364181b07ab7c7f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } } peers { permanent_uuid: "1167282fac594599842835302ef0d373" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } } }
I20260928 16:44:10.569881 15994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1167282fac594599842835302ef0d373 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:10.570393 15993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32160554974149ab9b0bc947acb589dd. Latest consensus state: current_term: 1 leader_uuid: "32160554974149ab9b0bc947acb589dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32160554974149ab9b0bc947acb589dd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39553 } } peers { permanent_uuid: "c7342e3be3cc4dc7a364181b07ab7c7f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } } peers { permanent_uuid: "1167282fac594599842835302ef0d373" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } } }
I20260928 16:44:10.570804 15993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:10.572570 16026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:10.575081 15753 master_runner.cc:263] No leader master found from master 32160554974149ab9b0bc947acb589dd
I20260928 16:44:10.575176 15753 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:44:10.575757 15902 master_runner.cc:263] No leader master found from master 1167282fac594599842835302ef0d373
I20260928 16:44:10.575843 15902 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:44:10.579615 16026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260928 16:44:10.580473 16015 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1167282fac594599842835302ef0d373: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:44:10.580600 16015 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:44:10.587136 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.568999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.25.71.125:46495)
0928 16:44:10.569167 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.569173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.575246 (+ 6073us) server_negotiation.cc:407] Connection header received
0928 16:44:10.575316 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.575320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.575387 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.575483 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.577864 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.578745 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.580400 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.580714 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.580795 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.583263 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:10.583267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:10.583271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:10.583302 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:10.585094 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.585628 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.585632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.585634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.585727 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.586027 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.586030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.586032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.586286 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:10.586398 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.586804 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.586949 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:44:10.588368 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.568256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58120 (local address 127.25.71.126:39553)
0928 16:44:10.568367 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.568371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.573628 (+ 5257us) server_negotiation.cc:407] Connection header received
0928 16:44:10.575614 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.575617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.575669 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.575717 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.577888 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.578729 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.580309 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.580631 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.580718 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.583256 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:10.583261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:10.583264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:10.583303 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:10.585453 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.586079 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.586084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.586086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.586197 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.586562 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.586566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.586568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.586893 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:10.587013 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.587965 (+ 952us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.588159 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 16:44:10.595631 15976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.567422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33276 (local address 127.25.71.124:38091)
0928 16:44:10.567562 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.567567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.575466 (+ 7899us) server_negotiation.cc:407] Connection header received
0928 16:44:10.575568 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.575571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.575621 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.575688 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.580762 (+ 5074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.581472 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.582657 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.583017 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.583100 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.590809 (+ 7709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:10.590813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:10.590817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:10.590876 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:10.593056 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.593777 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.593781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.593784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.593900 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.594291 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.594295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.594297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.594598 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:10.594804 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.595242 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.595409 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:44:10.602517 15945 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1167282fac594599842835302ef0d373 [term 1 FOLLOWER]: Refusing update from remote peer 32160554974149ab9b0bc947acb589dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:10.604153 16021 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1167282fac594599842835302ef0d373" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:10.610168 15994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1167282fac594599842835302ef0d373 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32160554974149ab9b0bc947acb589dd. Latest consensus state: current_term: 1 leader_uuid: "32160554974149ab9b0bc947acb589dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32160554974149ab9b0bc947acb589dd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39553 } } peers { permanent_uuid: "c7342e3be3cc4dc7a364181b07ab7c7f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } } peers { permanent_uuid: "1167282fac594599842835302ef0d373" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } } }
I20260928 16:44:10.610417 15994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1167282fac594599842835302ef0d373 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:10.611757 15870 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c7342e3be3cc4dc7a364181b07ab7c7f [term 1 FOLLOWER]: Refusing update from remote peer 32160554974149ab9b0bc947acb589dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:10.612766 16021 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7342e3be3cc4dc7a364181b07ab7c7f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:10.618563 15978 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7342e3be3cc4dc7a364181b07ab7c7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32160554974149ab9b0bc947acb589dd. Latest consensus state: current_term: 1 leader_uuid: "32160554974149ab9b0bc947acb589dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32160554974149ab9b0bc947acb589dd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39553 } } peers { permanent_uuid: "c7342e3be3cc4dc7a364181b07ab7c7f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } } peers { permanent_uuid: "1167282fac594599842835302ef0d373" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } } }
I20260928 16:44:10.618840 15978 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7342e3be3cc4dc7a364181b07ab7c7f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:10.625882 16021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32160554974149ab9b0bc947acb589dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32160554974149ab9b0bc947acb589dd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39553 } } peers { permanent_uuid: "c7342e3be3cc4dc7a364181b07ab7c7f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } } peers { permanent_uuid: "1167282fac594599842835302ef0d373" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } } }
I20260928 16:44:10.626111 16021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:10.628158 15994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1167282fac594599842835302ef0d373 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32160554974149ab9b0bc947acb589dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32160554974149ab9b0bc947acb589dd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39553 } } peers { permanent_uuid: "c7342e3be3cc4dc7a364181b07ab7c7f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } } peers { permanent_uuid: "1167282fac594599842835302ef0d373" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } } }
I20260928 16:44:10.628358 15994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1167282fac594599842835302ef0d373 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:10.628830 16029 mvcc.cc:205] Tried to move back new op lower bound from 7334354332055134208 to 7334354331930890240. Current Snapshot: MvccSnapshot[applied={T|T < 7334354332055134208}]
I20260928 16:44:10.629478 16030 mvcc.cc:205] Tried to move back new op lower bound from 7334354332055134208 to 7334354331930890240. Current Snapshot: MvccSnapshot[applied={T|T < 7334354332055134208}]
I20260928 16:44:10.635327 15993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32160554974149ab9b0bc947acb589dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32160554974149ab9b0bc947acb589dd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39553 } } peers { permanent_uuid: "c7342e3be3cc4dc7a364181b07ab7c7f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } } peers { permanent_uuid: "1167282fac594599842835302ef0d373" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } } }
I20260928 16:44:10.635550 15993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:10.637374 16032 mvcc.cc:205] Tried to move back new op lower bound from 7334354332055134208 to 7334354331930890240. Current Snapshot: MvccSnapshot[applied={T|T < 7334354332055134208}]
I20260928 16:44:10.637569 15978 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7342e3be3cc4dc7a364181b07ab7c7f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32160554974149ab9b0bc947acb589dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32160554974149ab9b0bc947acb589dd" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 39553 } } peers { permanent_uuid: "c7342e3be3cc4dc7a364181b07ab7c7f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 46495 } } peers { permanent_uuid: "1167282fac594599842835302ef0d373" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 38091 } } }
I20260928 16:44:10.637720 15978 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7342e3be3cc4dc7a364181b07ab7c7f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:10.641350 16026 catalog_manager.cc:1347] Generated new cluster ID: 402de790f14b4c18aeff8e96e13f04ff
I20260928 16:44:10.641481 16026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:10.673565 16026 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:10.675432 16026 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:10.704170 16026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32160554974149ab9b0bc947acb589dd: Generated new TSK 0
I20260928 16:44:10.705575 16026 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:44:10.746910 15824 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:10.743466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58130 (local address 127.25.71.126:39553)
0928 16:44:10.743616 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.743621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.745604 (+ 1983us) server_negotiation.cc:407] Connection header received
0928 16:44:10.745906 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.745910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.745969 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.746027 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.746741 (+ 714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58130: BlockingRecv error: recv got EOF from 127.0.0.1:58130 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 16:44:10.748662 15901 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:10.744283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.25.71.125:46495)
0928 16:44:10.744401 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.744405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.747321 (+ 2916us) server_negotiation.cc:407] Connection header received
0928 16:44:10.747663 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.747666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.747724 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.747775 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.748527 (+ 752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43012: BlockingRecv error: recv got EOF from 127.0.0.1:43012 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 16:44:10.749933 15976 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:10.744973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33286 (local address 127.25.71.124:38091)
0928 16:44:10.745086 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.745090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.748025 (+ 2935us) server_negotiation.cc:407] Connection header received
0928 16:44:10.749156 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.749160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.749223 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.749280 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.749788 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33286: BlockingRecv error: recv got EOF from 127.0.0.1:33286 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 16:44:10.755381 15901 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:10.753571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43024 (local address 127.25.71.125:46495)
0928 16:44:10.753689 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.753694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.754211 (+ 517us) server_negotiation.cc:407] Connection header received
0928 16:44:10.754494 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.754497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.754570 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.754698 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.755253 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43024: BlockingRecv error: recv got EOF from 127.0.0.1:43024 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 16:44:10.761133 15901 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:10.759109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43028 (local address 127.25.71.125:46495)
0928 16:44:10.759253 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.759258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.759833 (+ 575us) server_negotiation.cc:407] Connection header received
0928 16:44:10.760033 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.760036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.760104 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.760244 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.761004 (+ 760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43028: BlockingRecv error: recv got EOF from 127.0.0.1:43028 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 16:44:10.767019 15901 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:10.765515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43040 (local address 127.25.71.125:46495)
0928 16:44:10.765652 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.765657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.766146 (+ 489us) server_negotiation.cc:407] Connection header received
0928 16:44:10.766283 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.766286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.766338 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.766405 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.766908 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43040: BlockingRecv error: recv got EOF from 127.0.0.1:43040 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 16:44:10.774521 15901 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:10.773389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43048 (local address 127.25.71.125:46495)
0928 16:44:10.773577 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.773582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.773727 (+ 145us) server_negotiation.cc:407] Connection header received
0928 16:44:10.773871 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.773875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.773923 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.773991 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.774420 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43048: BlockingRecv error: recv got EOF from 127.0.0.1:43048 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260928 16:44:10.786514 15901 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:10.785284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43054 (local address 127.25.71.125:46495)
0928 16:44:10.785448 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.785453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.785655 (+ 202us) server_negotiation.cc:407] Connection header received
0928 16:44:10.785800 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.785802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.785849 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.785915 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.786383 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43054: BlockingRecv error: recv got EOF from 127.0.0.1:43054 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 16:44:10.806219 15901 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:10.805027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43058 (local address 127.25.71.125:46495)
0928 16:44:10.805164 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.805168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.805350 (+ 182us) server_negotiation.cc:407] Connection header received
0928 16:44:10.805503 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.805506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.805553 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.805622 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.806088 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43058: BlockingRecv error: recv got EOF from 127.0.0.1:43058 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 16:44:10.842581 15901 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:10.841063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43072 (local address 127.25.71.125:46495)
0928 16:44:10.841194 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.841199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.841413 (+ 214us) server_negotiation.cc:407] Connection header received
0928 16:44:10.841640 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.841643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.841703 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.841799 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.842461 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43072: BlockingRecv error: recv got EOF from 127.0.0.1:43072 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 16:44:10.910835 15901 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:10.909381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.25.71.125:46495)
0928 16:44:10.909562 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.909567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.909807 (+ 240us) server_negotiation.cc:407] Connection header received
0928 16:44:10.909967 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.909970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.910065 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.910141 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.910719 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43086: BlockingRecv error: recv got EOF from 127.0.0.1:43086 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260928 16:44:11.043437 15901 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:11.041863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43092 (local address 127.25.71.125:46495)
0928 16:44:11.042042 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:11.042048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:11.042311 (+ 263us) server_negotiation.cc:407] Connection header received
0928 16:44:11.042520 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:11.042523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:11.042583 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:11.042727 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:11.043267 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43092: BlockingRecv error: recv got EOF from 127.0.0.1:43092 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260928 16:44:11.303936 15901 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:11.302396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43094 (local address 127.25.71.125:46495)
0928 16:44:11.302534 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:11.302539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:11.302558 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:44:11.302897 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:11.302900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:11.302956 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:11.303044 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:11.303729 (+ 685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43094: BlockingRecv error: recv got EOF from 127.0.0.1:43094 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:44:11.548452 15990 catalog_manager.cc:1259] Loaded cluster ID: 402de790f14b4c18aeff8e96e13f04ff
I20260928 16:44:11.548552 15990 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c7342e3be3cc4dc7a364181b07ab7c7f: loading cluster ID for follower catalog manager: success
I20260928 16:44:11.550341 15990 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c7342e3be3cc4dc7a364181b07ab7c7f: acquiring CA information for follower catalog manager: success
I20260928 16:44:11.551764 15990 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c7342e3be3cc4dc7a364181b07ab7c7f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:44:11.582334 16015 catalog_manager.cc:1259] Loaded cluster ID: 402de790f14b4c18aeff8e96e13f04ff
I20260928 16:44:11.582436 16015 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1167282fac594599842835302ef0d373: loading cluster ID for follower catalog manager: success
I20260928 16:44:11.584318 16015 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1167282fac594599842835302ef0d373: acquiring CA information for follower catalog manager: success
I20260928 16:44:11.586148 16015 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1167282fac594599842835302ef0d373: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:44:11.823793 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:11.818843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43098 (local address 127.25.71.125:46495)
0928 16:44:11.819158 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:11.819162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:11.819182 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:44:11.819244 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:11.819247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:11.819382 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:11.819525 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:11.819901 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:11.820487 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:11.821688 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:11.821999 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:11.822153 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:11.822696 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:11.823242 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:11.823369 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:11.823558 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 16:44:11.823636 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":124,"threads_started":1}
I20260928 16:44:11.832930 16108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:11.828480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33298 (local address 127.25.71.124:38091)
0928 16:44:11.828735 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:11.828739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:11.828755 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:44:11.828962 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:11.828965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:11.829112 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:11.829245 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:11.829533 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:11.830122 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:11.831191 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:11.831432 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:11.831523 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:11.831940 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:11.832474 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:11.832585 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:11.832723 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 16:44:11.832790 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":90,"threads_started":1}
I20260928 16:44:11.842701 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:11.837917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43112 (local address 127.25.71.125:46495)
0928 16:44:11.838053 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:11.838057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:11.838611 (+ 554us) server_negotiation.cc:407] Connection header received
0928 16:44:11.838749 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:11.838752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:11.838813 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:11.838938 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:11.839236 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:11.840105 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:11.841278 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:11.841542 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:11.841647 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:11.842094 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:11.842195 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:11.842263 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:11.842422 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 16:44:11.842503 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 16:44:11.843310 16108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:11.837601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33306 (local address 127.25.71.124:38091)
0928 16:44:11.837792 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:11.837798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:11.838388 (+ 590us) server_negotiation.cc:407] Connection header received
0928 16:44:11.838736 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:11.838739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:11.838800 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:11.838854 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:11.839386 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:11.840162 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:11.841679 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:11.841934 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:11.842044 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:11.842674 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:11.842803 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:11.842881 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:11.843040 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 16:44:11.843134 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 16:44:11.843475 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:11.837373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58144 (local address 127.25.71.126:39553)
0928 16:44:11.838137 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:11.838143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:11.838417 (+ 274us) server_negotiation.cc:407] Connection header received
0928 16:44:11.838600 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:11.838604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:11.838878 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:11.839112 (+ 234us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:11.839502 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:11.840193 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:11.841660 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:11.841996 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:11.842099 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:11.842425 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:11.842926 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:11.843017 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:11.843205 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 16:44:11.843312 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":123,"threads_started":1}
I20260928 16:44:11.848398 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 15753
I20260928 16:44:11.856068 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 15828
I20260928 16:44:11.864727 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 15902
2026-09-28T16:44:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3214 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:11 dist-test-slave-nqgh krb5kdc[16119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:11 dist-test-slave-nqgh krb5kdc[16119](info): set up 2 sockets
Sep 28 16:44:11 dist-test-slave-nqgh krb5kdc[16119](info): commencing operation
krb5kdc: starting...
W20260928 16:44:12.853390 25885 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/test-user.keytab.
Sep 28 16:44:12 dist-test-slave-nqgh krb5kdc[16119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
Entry for principal kudu/127.25.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab.
I20260928 16:44:12.999416 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:35635
--webserver_interface=127.25.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:35635,127.25.71.125:38147,127.25.71.124:36995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:13.121893 16132 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:44:13.122257 16132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:13.122293 16132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:13.122355 16132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:13.127501 16132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:13.127614 16132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:13.127647 16132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:13.127669 16132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:13.127689 16132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:13.134372 16132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:35635,127.25.71.125:38147,127.25.71.124:36995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.126.keytab
--principal=kudu/127.25.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.126:35635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.71.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:13.135680 16132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:13.136797 16132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:13.147578 16140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:13.147639 16138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:13.147612 16137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:13.148156 16132 server_base.cc:1034] running on GCE node
I20260928 16:44:13.148888 16132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:13.150456 16132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:13.153873 16132 hybrid_clock.cc:648] HybridClock initialized: now 1790613853153818 us; error 65 us; skew 500 ppm
Sep 28 16:44:13 dist-test-slave-nqgh krb5kdc[16119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613853, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:13.157691 16132 init.cc:377] Logged in from keytab as kudu/127.25.71.126@KRBTEST.COM (short username kudu)
I20260928 16:44:13.159221 16132 webserver.cc:460] Webserver started at http://127.25.71.126:40517/ using document root <none> and password file <none>
I20260928 16:44:13.159909 16132 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:13.160007 16132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:13.160252 16132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:13.162675 16132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-0/data/instance:
uuid: "871ed195a0d44e6d83ed184615b68822"
format_stamp: "Formatted at 2026-09-28 16:44:13 on dist-test-slave-nqgh"
server_key: "174e61f9ab3f2194080d86a4e584818a"
server_key_iv: "a58795439cdb85f29b9b7cfe42abb6e3"
server_key_version: "encryptionkey@0"
I20260928 16:44:13.163341 16132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-0/wal/instance:
uuid: "871ed195a0d44e6d83ed184615b68822"
format_stamp: "Formatted at 2026-09-28 16:44:13 on dist-test-slave-nqgh"
server_key: "174e61f9ab3f2194080d86a4e584818a"
server_key_iv: "a58795439cdb85f29b9b7cfe42abb6e3"
server_key_version: "encryptionkey@0"
I20260928 16:44:13.168066 16132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:44:13.171345 16147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:13.173235 16132 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:44:13.173425 16132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-0/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-0/wal
uuid: "871ed195a0d44e6d83ed184615b68822"
format_stamp: "Formatted at 2026-09-28 16:44:13 on dist-test-slave-nqgh"
server_key: "174e61f9ab3f2194080d86a4e584818a"
server_key_iv: "a58795439cdb85f29b9b7cfe42abb6e3"
server_key_version: "encryptionkey@0"
I20260928 16:44:13.173563 16132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:13.185401 16132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:13.188879 16132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:13.189119 16132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:13.197261 16132 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.126:35635
I20260928 16:44:13.197656 16199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.126:35635 every 8 connection(s)
I20260928 16:44:13.198699 16132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:13.202975 16200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:13.208803 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 16132
I20260928 16:44:13.208381 16200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35635 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:44:13.208956 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-0/wal/instance
I20260928 16:44:13.209340 25885 external_mini_cluster.cc:1428] Setting key 3d644bd381150bbe2227ac8ecfaeaba0
WARNING: no policy specified for kudu/127.25.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.125@KRBTEST.COM" created.
Sep 28 16:44:13 dist-test-slave-nqgh krb5kdc[16119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613853, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:44:13.231508 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.212365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35635 (local address 127.0.0.1:42594)
0928 16:44:13.213146 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.213194 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:13.214349 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:13.215264 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:13.215280 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:13.215830 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:13.216129 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.216145 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.217436 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.217439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.218326 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.218337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.219445 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.219450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.219575 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.220379 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:13.220402 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:13.223180 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:13.226670 (+ 3490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.226707 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.226720 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.227156 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.227786 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.227789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.227790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.227988 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.229310 (+ 1322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:13.229318 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:13.229556 (+ 238us) client_negotiation.cc:769] Sending connection context
0928 16:44:13.229767 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 16:44:13.230173 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"spinlock_wait_cycles":896,"thread_start_us":256,"threads_started":1}
I20260928 16:44:13.231504 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.212376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42594 (local address 127.25.71.126:35635)
0928 16:44:13.213010 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:13.213032 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:13.213340 (+ 308us) server_negotiation.cc:407] Connection header received
0928 16:44:13.214777 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:13.214782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:13.215042 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:13.215242 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:13.216295 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.217296 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.218443 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.219135 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.219471 (+ 336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.223540 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:13.223552 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:13.223572 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:13.223630 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:13.226371 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.227346 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.227357 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.227359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.227605 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.228228 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.228232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.228234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.229068 (+ 834us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:13.229297 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:13.229917 (+ 620us) server_negotiation.cc:299] Negotiation successful
0928 16:44:13.230350 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":128,"server-negotiator.queue_time_us":451,"thread_start_us":196,"threads_started":1}
I20260928 16:44:13.235351 16200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:44:13.236872 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.236394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38147 (local address 127.0.0.1:43402)
0928 16:44:13.236626 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.236754 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260928 16:44:13.237228 16148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38147: connect: Connection refused (error 111)
I20260928 16:44:13.238794 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.238437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38147 (local address 127.0.0.1:43412)
0928 16:44:13.238584 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.238687 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 16:44:13.239073 16200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.125:38147: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38147: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.125.keytab.
Entry for principal kudu/127.25.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.125.keytab.
I20260928 16:44:13.243935 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.125:38147
--webserver_interface=127.25.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.125.keytab
--principal=kudu/127.25.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:35635,127.25.71.125:38147,127.25.71.124:36995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:44:13.288687 16200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } attempt: 1
I20260928 16:44:13.289932 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.289325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38147 (local address 127.0.0.1:43418)
0928 16:44:13.289603 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.289783 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 16:44:13.290966 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.290656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38147 (local address 127.0.0.1:43434)
0928 16:44:13.290783 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.290869 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 16:44:13.291222 16200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.125:38147: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38147: connect: Connection refused (error 111)
I20260928 16:44:13.359578 16200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } attempt: 2
I20260928 16:44:13.360824 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.360219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38147 (local address 127.0.0.1:43436)
0928 16:44:13.360550 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.360678 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260928 16:44:13.361914 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.361574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38147 (local address 127.0.0.1:43440)
0928 16:44:13.361732 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.361817 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 16:44:13.362170 16200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.125:38147: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38147: connect: Connection refused (error 111)
W20260928 16:44:13.392306 16207 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:44:13.392758 16207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:13.392786 16207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:13.392833 16207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:13.397714 16207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:13.397833 16207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:13.397855 16207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:13.397872 16207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:13.397951 16207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:13.404790 16207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:35635,127.25.71.125:38147,127.25.71.124:36995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.125.keytab
--principal=kudu/127.25.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.125:38147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.71.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:13.406401 16207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:13.407855 16207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:13.419086 16212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:13.419193 16213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:13.419096 16215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:13.419167 16207 server_base.cc:1034] running on GCE node
I20260928 16:44:13.420195 16207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:13.421561 16207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:13.422889 16207 hybrid_clock.cc:648] HybridClock initialized: now 1790613853422860 us; error 58 us; skew 500 ppm
Sep 28 16:44:13 dist-test-slave-nqgh krb5kdc[16119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613853, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:13.426820 16207 init.cc:377] Logged in from keytab as kudu/127.25.71.125@KRBTEST.COM (short username kudu)
I20260928 16:44:13.428068 16207 webserver.cc:460] Webserver started at http://127.25.71.125:38391/ using document root <none> and password file <none>
I20260928 16:44:13.428720 16207 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:13.428800 16207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:13.429028 16207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:13.431466 16207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-1/data/instance:
uuid: "971f17b386f64ffe830316662ec38f0f"
format_stamp: "Formatted at 2026-09-28 16:44:13 on dist-test-slave-nqgh"
server_key: "6d777ae84c6154da91b47c810c336c5f"
server_key_iv: "abe79a7d8d5a6b5eae226ce20408ed39"
server_key_version: "encryptionkey@0"
I20260928 16:44:13.432156 16207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-1/wal/instance:
uuid: "971f17b386f64ffe830316662ec38f0f"
format_stamp: "Formatted at 2026-09-28 16:44:13 on dist-test-slave-nqgh"
server_key: "6d777ae84c6154da91b47c810c336c5f"
server_key_iv: "abe79a7d8d5a6b5eae226ce20408ed39"
server_key_version: "encryptionkey@0"
I20260928 16:44:13.437312 16207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:44:13.441646 16222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:13.443738 16207 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:44:13.443943 16207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-1/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-1/wal
uuid: "971f17b386f64ffe830316662ec38f0f"
format_stamp: "Formatted at 2026-09-28 16:44:13 on dist-test-slave-nqgh"
server_key: "6d777ae84c6154da91b47c810c336c5f"
server_key_iv: "abe79a7d8d5a6b5eae226ce20408ed39"
server_key_version: "encryptionkey@0"
I20260928 16:44:13.444095 16207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:13.453357 16200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } attempt: 3
I20260928 16:44:13.454703 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.454172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38147 (local address 127.0.0.1:43456)
0928 16:44:13.454410 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.454507 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 16:44:13.455606 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.455326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38147 (local address 127.0.0.1:43462)
0928 16:44:13.455440 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.455520 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 16:44:13.455613 16207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
W20260928 16:44:13.455806 16200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.125:38147: Network error: Client connection negotiation failed: client connection to 127.25.71.125:38147: connect: Connection refused (error 111)
I20260928 16:44:13.459772 16207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:13.459967 16207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:13.468371 16207 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.125:38147
I20260928 16:44:13.468744 16274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.125:38147 every 8 connection(s)
I20260928 16:44:13.469787 16207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-1/data/info.pb
I20260928 16:44:13.473861 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 16207
I20260928 16:44:13.474007 16275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:13.474047 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-1/wal/instance
I20260928 16:44:13.474453 25885 external_mini_cluster.cc:1428] Setting key 475d50c2664b7ef0bb9e56ab26194675
I20260928 16:44:13.480372 16275 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35635 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.71.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.71.124@KRBTEST.COM" created.
Sep 28 16:44:13 dist-test-slave-nqgh krb5kdc[16119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613853, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.125@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:44:13.498494 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.483100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42604 (local address 127.25.71.126:35635)
0928 16:44:13.483337 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:13.483343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:13.484695 (+ 1352us) server_negotiation.cc:407] Connection header received
0928 16:44:13.485904 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:13.485909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:13.485981 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:13.486059 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:13.487635 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.488204 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.489474 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.489717 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.489811 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.493468 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:13.493473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:13.493475 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:13.493525 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:13.495165 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.496144 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.496148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.496149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.496261 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.496811 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.496815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.496816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.497154 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:13.497267 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:13.498048 (+ 781us) server_negotiation.cc:299] Negotiation successful
0928 16:44:13.498299 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 16:44:13.499312 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.483608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35635 (local address 127.0.0.1:42604)
0928 16:44:13.484441 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.484512 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:13.485644 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:13.486278 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:13.486286 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:13.486750 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:13.487454 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.487473 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.488346 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.488349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.489335 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.489346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.489902 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.489907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.490075 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.490782 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:13.490803 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:13.493113 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:13.495370 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.495379 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.495403 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.495886 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.496392 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.496395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.496397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.496636 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.497315 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:13.497324 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:13.497694 (+ 370us) client_negotiation.cc:769] Sending connection context
0928 16:44:13.497944 (+ 250us) client_negotiation.cc:240] Negotiation successful
0928 16:44:13.498285 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":205,"threads_started":1}
I20260928 16:44:13.501483 16275 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.124.keytab.
Entry for principal kudu/127.25.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.124.keytab.
I20260928 16:44:13.507972 25885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskp76gbp/build/debug/bin/kudu
/tmp/dist-test-taskp76gbp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.124:36995
--webserver_interface=127.25.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.124.keytab
--principal=kudu/127.25.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.71.84:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.71.126:35635,127.25.71.125:38147,127.25.71.124:36995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:44:13 dist-test-slave-nqgh krb5kdc[16119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613853, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.125@KRBTEST.COM for kudu/127.25.71.125@KRBTEST.COM
I20260928 16:44:13.513866 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.502296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38147 (local address 127.0.0.1:43464)
0928 16:44:13.502480 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.502498 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:13.502734 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:13.503508 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:13.503510 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:13.503725 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:13.503891 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.503900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.505196 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.505199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.505889 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.505901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.506771 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.506774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.506901 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.507604 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:13.507620 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:13.509105 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:13.511595 (+ 2490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.511600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.511602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.511924 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.512364 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.512368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.512370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.512485 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.513087 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:13.513090 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:13.513250 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 16:44:13.513381 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 16:44:13.513587 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 16:44:13.513900 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.502532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43464 (local address 127.25.71.125:38147)
0928 16:44:13.502987 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:13.502998 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:13.503026 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:44:13.503091 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:13.503096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:13.503324 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:13.503480 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:13.504064 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.505029 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.506064 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.506541 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.506743 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.509400 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:13.509414 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:13.509440 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:13.509478 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:13.511427 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.512110 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.512117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.512118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.512237 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.512621 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.512624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.512625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.512958 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:13.513089 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:13.513473 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 16:44:13.513650 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":181,"threads_started":1}
I20260928 16:44:13.517032 16275 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:44:13.518158 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.517741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:36995 (local address 127.0.0.1:37766)
0928 16:44:13.517925 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.518050 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260928 16:44:13.518505 16226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
I20260928 16:44:13.519624 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.519384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:36995 (local address 127.0.0.1:37778)
0928 16:44:13.519500 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.519554 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 16:44:13.519768 16275 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.124:36995: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
I20260928 16:44:13.569097 16275 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } attempt: 1
I20260928 16:44:13.570017 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.569714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:36995 (local address 127.0.0.1:37784)
0928 16:44:13.569882 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.569938 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 16:44:13.571204 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.570921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:36995 (local address 127.0.0.1:37786)
0928 16:44:13.571079 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.571140 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260928 16:44:13.571355 16275 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.124:36995: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
I20260928 16:44:13.599049 16200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } attempt: 4
Sep 28 16:44:13 dist-test-slave-nqgh krb5kdc[16119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613853, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for kudu/127.25.71.125@KRBTEST.COM
I20260928 16:44:13.608412 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.599844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38147 (local address 127.0.0.1:43474)
0928 16:44:13.600096 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.600115 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:13.600259 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:13.600558 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:13.600566 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:13.600766 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:13.600958 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.600966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.601744 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.601748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.602282 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.602290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.602839 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.602850 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.602964 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.603647 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:13.603660 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:13.605100 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:13.606945 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.606957 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.606959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.607162 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.607408 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.607411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.607412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.607490 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.607854 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:13.607857 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:13.607967 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 16:44:13.608067 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 16:44:13.608228 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260928 16:44:13.608412 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.599944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43474 (local address 127.25.71.125:38147)
0928 16:44:13.600096 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:13.600100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:13.600203 (+ 103us) server_negotiation.cc:407] Connection header received
0928 16:44:13.600388 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:13.600391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:13.600437 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:13.600507 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:13.601080 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.601641 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.602406 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.602707 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.602813 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.605254 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:13.605257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:13.605259 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:13.605284 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:13.606826 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.607253 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.607256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.607258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.607328 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.607570 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.607573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.607575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.607767 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:13.607850 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:13.608138 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 16:44:13.608255 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 16:44:13.610208 16200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:44:13.611102 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.610779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:36995 (local address 127.0.0.1:37788)
0928 16:44:13.610976 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.611029 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 16:44:13.612068 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.611812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:36995 (local address 127.0.0.1:37800)
0928 16:44:13.611948 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.611997 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 16:44:13.612203 16200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.124:36995: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
I20260928 16:44:13.639618 16275 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } attempt: 2
I20260928 16:44:13.640918 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.640462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:36995 (local address 127.0.0.1:37808)
0928 16:44:13.640727 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.640817 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260928 16:44:13.642011 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.641644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:36995 (local address 127.0.0.1:37824)
0928 16:44:13.641806 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.641885 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260928 16:44:13.642217 16275 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.124:36995: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
W20260928 16:44:13.651899 16281 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:44:13.652376 16281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:13.652412 16281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:13.652472 16281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:13.657557 16281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:13.657688 16281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:13.657709 16281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:13.657727 16281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:13.657743 16281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:13.665154 16281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.71.84:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.71.126:35635,127.25.71.125:38147,127.25.71.124:36995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/krb5kdc/kudu_127.25.71.124.keytab
--principal=kudu/127.25.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.71.124:36995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.71.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:13.666883 16281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:13.668272 16281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:44:13.671420 16200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } attempt: 1
I20260928 16:44:13.672298 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.671989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:36995 (local address 127.0.0.1:37828)
0928 16:44:13.672139 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.672198 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 16:44:13.673482 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.673171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:36995 (local address 127.0.0.1:37834)
0928 16:44:13.673347 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.673414 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 16:44:13.673673 16200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.124:36995: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
W20260928 16:44:13.679214 16288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:13.679150 16287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:13.679198 16290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:13.679962 16281 server_base.cc:1034] running on GCE node
I20260928 16:44:13.680836 16281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:13.682530 16281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:13.683799 16281 hybrid_clock.cc:648] HybridClock initialized: now 1790613853683758 us; error 65 us; skew 500 ppm
Sep 28 16:44:13 dist-test-slave-nqgh krb5kdc[16119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613853, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:13.688722 16281 init.cc:377] Logged in from keytab as kudu/127.25.71.124@KRBTEST.COM (short username kudu)
I20260928 16:44:13.690387 16281 webserver.cc:460] Webserver started at http://127.25.71.124:42691/ using document root <none> and password file <none>
I20260928 16:44:13.691304 16281 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:13.691406 16281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:13.691687 16281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:13.694612 16281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-2/data/instance:
uuid: "300ada7ae9854b1882898fcafa86c4ae"
format_stamp: "Formatted at 2026-09-28 16:44:13 on dist-test-slave-nqgh"
server_key: "575f7ccfd6f8b648f127b3b79fff7948"
server_key_iv: "5a7319e385ecee3253b5591ba5b72931"
server_key_version: "encryptionkey@0"
I20260928 16:44:13.695536 16281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-2/wal/instance:
uuid: "300ada7ae9854b1882898fcafa86c4ae"
format_stamp: "Formatted at 2026-09-28 16:44:13 on dist-test-slave-nqgh"
server_key: "575f7ccfd6f8b648f127b3b79fff7948"
server_key_iv: "5a7319e385ecee3253b5591ba5b72931"
server_key_version: "encryptionkey@0"
I20260928 16:44:13.701273 16281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:44:13.705174 16297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:13.707321 16281 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:44:13.707544 16281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-2/data,/tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-2/wal
uuid: "300ada7ae9854b1882898fcafa86c4ae"
format_stamp: "Formatted at 2026-09-28 16:44:13 on dist-test-slave-nqgh"
server_key: "575f7ccfd6f8b648f127b3b79fff7948"
server_key_iv: "5a7319e385ecee3253b5591ba5b72931"
server_key_version: "encryptionkey@0"
I20260928 16:44:13.707727 16281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:13.733428 16275 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } attempt: 3
I20260928 16:44:13.734550 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.734162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:36995 (local address 127.0.0.1:37838)
0928 16:44:13.734373 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.734459 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 16:44:13.735015 16281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:13.735651 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.735160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:36995 (local address 127.0.0.1:37846)
0928 16:44:13.735388 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.735471 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260928 16:44:13.735970 16275 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.124:36995: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
I20260928 16:44:13.739338 16281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:13.739570 16281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:13.740860 16200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } attempt: 2
I20260928 16:44:13.741923 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.741458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:36995 (local address 127.0.0.1:37848)
0928 16:44:13.741696 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.741800 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 16:44:13.742985 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.742688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:36995 (local address 127.0.0.1:37850)
0928 16:44:13.742850 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.742913 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260928 16:44:13.743142 16200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.71.124:36995: Network error: Client connection negotiation failed: client connection to 127.25.71.124:36995: connect: Connection refused (error 111)
I20260928 16:44:13.748206 16281 rpc_server.cc:307] RPC server started. Bound to: 127.25.71.124:36995
I20260928 16:44:13.748584 16349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.71.124:36995 every 8 connection(s)
I20260928 16:44:13.749413 16281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-2/data/info.pb
I20260928 16:44:13.753547 16350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:13.757545 25885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskp76gbp/build/debug/bin/kudu as pid 16281
I20260928 16:44:13.757684 25885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskp76gbp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613545667819-25885-0/minicluster-data/master-2/wal/instance
I20260928 16:44:13.758052 25885 external_mini_cluster.cc:1428] Setting key 7d7556e5fcd29c62db0d999db5d55362
I20260928 16:44:13.759298 16350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35635 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:44:13 dist-test-slave-nqgh krb5kdc[16119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613853, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.124@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
Sep 28 16:44:13 dist-test-slave-nqgh krb5kdc[16119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.124@KRBTEST.COM
I20260928 16:44:13.774809 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.761972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42614 (local address 127.25.71.126:35635)
0928 16:44:13.762160 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:13.762166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:13.762607 (+ 441us) server_negotiation.cc:407] Connection header received
0928 16:44:13.762896 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:13.762900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:13.762952 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:13.763029 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:13.764064 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.764596 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.765825 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.766116 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.766229 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.770111 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:13.770116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:13.770119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:13.770161 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:13.772002 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.772756 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.772761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.772763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.772892 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.773332 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.773336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.773338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.773590 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:13.773686 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:13.774436 (+ 750us) server_negotiation.cc:299] Negotiation successful
0928 16:44:13.774598 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 16:44:13.776554 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.759985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37864 (local address 127.25.71.124:36995)
0928 16:44:13.760571 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:13.760596 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:13.760666 (+ 70us) server_negotiation.cc:407] Connection header received
0928 16:44:13.761628 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:13.761651 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:13.762155 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:13.762566 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:13.763598 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.764809 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.765944 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.766581 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.767338 (+ 757us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.770400 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:13.770409 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:13.770430 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:13.770460 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:13.772417 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.773001 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.773011 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.773013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.773205 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.773963 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.773967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.773969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.774255 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:13.775427 (+ 1172us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:13.775536 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 16:44:13.775671 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":379,"thread_start_us":153,"threads_started":1}
I20260928 16:44:13.776554 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.761883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.126:35635 (local address 127.0.0.1:42614)
0928 16:44:13.762464 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.762524 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:13.762759 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:13.763118 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:13.763125 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:13.763515 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:13.763910 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.763925 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.764713 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.764715 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.765642 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.765657 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.766321 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.766325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.766590 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.767387 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:13.767413 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:13.769835 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:13.772167 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.772181 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.772198 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.772604 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.773022 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.773025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.773027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.773220 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.773699 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:13.773707 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:13.774116 (+ 409us) client_negotiation.cc:769] Sending connection context
0928 16:44:13.775033 (+ 917us) client_negotiation.cc:240] Negotiation successful
0928 16:44:13.775451 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":42,"thread_start_us":129,"threads_started":1}
I20260928 16:44:13.779132 16350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:44:13 dist-test-slave-nqgh krb5kdc[16119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613853, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.124@KRBTEST.COM for kudu/127.25.71.125@KRBTEST.COM
Sep 28 16:44:13 dist-test-slave-nqgh krb5kdc[16119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.126@KRBTEST.COM
I20260928 16:44:13.790566 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.780228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43490 (local address 127.25.71.125:38147)
0928 16:44:13.780422 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:13.780427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:13.780452 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:44:13.780596 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:13.780599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:13.780651 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:13.780733 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:13.781355 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.782090 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.783202 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.783529 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.783647 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.786149 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:13.786154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:13.786158 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:13.786193 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:13.788230 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.788894 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.788898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.788899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.789061 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.789387 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.789391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.789393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.789675 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:13.789779 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:13.790190 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 16:44:13.790369 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:44:13.790723 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.780052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.125:38147 (local address 127.0.0.1:43490)
0928 16:44:13.780390 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.780404 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:13.780490 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:13.780789 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:13.780793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:13.781029 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:13.781195 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.781204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.782219 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.782223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.783052 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.783063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.783732 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.783736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.783882 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.784332 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:13.784345 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:13.785899 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:13.788419 (+ 2520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.788423 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.788425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.788744 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.789178 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.789181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.789183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.789282 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.789796 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:13.789798 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:13.789942 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 16:44:13.790066 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 16:44:13.790260 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251}
I20260928 16:44:13.791158 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.779328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42626 (local address 127.25.71.126:35635)
0928 16:44:13.779544 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:13.779549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:13.779566 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:44:13.779611 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:13.779614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:13.779676 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:13.779765 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:13.780757 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.781611 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.782667 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.783044 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.783179 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.786840 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:13.786845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:13.786848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:13.786887 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:13.788585 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.789204 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.789207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.789208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.789292 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.789592 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.789596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.789598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.789837 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:13.790566 (+ 729us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:13.790685 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 16:44:13.790818 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260928 16:44:13.792022 16350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:44:13 dist-test-slave-nqgh krb5kdc[16119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613853, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.124@KRBTEST.COM for kudu/127.25.71.124@KRBTEST.COM
Sep 28 16:44:13 dist-test-slave-nqgh krb5kdc[16119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.71.125@KRBTEST.COM
I20260928 16:44:13.803704 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.792849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:36995 (local address 127.0.0.1:37866)
0928 16:44:13.793063 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.793080 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:13.793168 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:13.793444 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:13.793447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:13.793681 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:13.793830 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.793838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.795125 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.795128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.796163 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.796176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.796979 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.796984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.797069 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.797701 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:13.797714 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:13.799540 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:13.801538 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.801543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.801545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.801831 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.802177 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.802181 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.802183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.802291 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.802963 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:13.802966 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:13.803104 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 16:44:13.803206 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 16:44:13.803410 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260928 16:44:13.803884 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.793080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.25.71.125:38147)
0928 16:44:13.793192 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:13.793196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:13.793207 (+ 11us) server_negotiation.cc:407] Connection header received
0928 16:44:13.793253 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:13.793255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:13.793305 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:13.793406 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:13.794229 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.795056 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.796378 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.796735 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.796866 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.800395 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:13.800400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:13.800403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:13.800442 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:13.801892 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.802492 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.802495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.802497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.802585 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.802983 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.802985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.802987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.803200 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:13.803529 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:13.803607 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 16:44:13.803734 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 16:44:13.803745 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.792930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37866 (local address 127.25.71.124:36995)
0928 16:44:13.793097 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:13.793101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:13.793122 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:44:13.793273 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:13.793276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:13.793323 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:13.793414 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:13.794010 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.794888 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.796372 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.796791 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.796977 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.799727 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:13.799730 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:13.799732 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:13.799754 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:13.801336 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.801947 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.801950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.801952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.802050 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.802463 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.802467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.802468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.802847 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:13.802934 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:13.803290 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 16:44:13.803510 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":63}
I20260928 16:44:13.808503 16350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 300ada7ae9854b1882898fcafa86c4ae: Bootstrap starting.
I20260928 16:44:13.809005 25885 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:44:13.811929 16350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 300ada7ae9854b1882898fcafa86c4ae: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:13.811220 16174 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "871ed195a0d44e6d83ed184615b68822"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42626
I20260928 16:44:13.812937 16350 log.cc:824] T 00000000000000000000000000000000 P 300ada7ae9854b1882898fcafa86c4ae: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:13.815925 16350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 300ada7ae9854b1882898fcafa86c4ae: No bootstrap required, opened a new log
I20260928 16:44:13.820766 16350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 300ada7ae9854b1882898fcafa86c4ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871ed195a0d44e6d83ed184615b68822" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35635 } } peers { permanent_uuid: "971f17b386f64ffe830316662ec38f0f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } } peers { permanent_uuid: "300ada7ae9854b1882898fcafa86c4ae" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } }
I20260928 16:44:13.821138 16350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 300ada7ae9854b1882898fcafa86c4ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 300ada7ae9854b1882898fcafa86c4ae, State: Initialized, Role: FOLLOWER
I20260928 16:44:13.821522 16350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 300ada7ae9854b1882898fcafa86c4ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871ed195a0d44e6d83ed184615b68822" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35635 } } peers { permanent_uuid: "971f17b386f64ffe830316662ec38f0f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } } peers { permanent_uuid: "300ada7ae9854b1882898fcafa86c4ae" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } }
I20260928 16:44:13.822398 16356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 300ada7ae9854b1882898fcafa86c4ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871ed195a0d44e6d83ed184615b68822" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35635 } } peers { permanent_uuid: "971f17b386f64ffe830316662ec38f0f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } } peers { permanent_uuid: "300ada7ae9854b1882898fcafa86c4ae" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } } }
I20260928 16:44:13.822674 16356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 300ada7ae9854b1882898fcafa86c4ae [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:13.823302 16350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 300ada7ae9854b1882898fcafa86c4ae [sys.catalog]: configured and running, proceeding with master startup.
W20260928 16:44:13.830525 16367 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 300ada7ae9854b1882898fcafa86c4ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:44:13.830736 16367 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:44:13.828847 16281 master_runner.cc:186] Error getting master registration for 127.25.71.126:35635: OK, instance_id { permanent_uuid: "871ed195a0d44e6d83ed184615b68822" instance_seqno: 1790613853194993 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:44:13.832491 16281 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:44:13.843312 16200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } attempt: 3
Sep 28 16:44:13 dist-test-slave-nqgh krb5kdc[16119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613853, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.126@KRBTEST.COM for kudu/127.25.71.124@KRBTEST.COM
I20260928 16:44:13.854558 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.844162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37880 (local address 127.25.71.124:36995)
0928 16:44:13.844317 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:13.844323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:13.844435 (+ 112us) server_negotiation.cc:407] Connection header received
0928 16:44:13.844599 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:13.844602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:13.844678 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:13.844793 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:13.845648 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.846470 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.847838 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.848199 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.848350 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.851185 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:13.851188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:13.851191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:13.851226 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:13.852670 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.853189 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.853193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.853195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.853282 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.853535 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.853538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.853540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.853836 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:13.853930 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:13.854212 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 16:44:13.854338 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:44:13.854548 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.843976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:36995 (local address 127.0.0.1:37880)
0928 16:44:13.844245 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.844270 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:13.844469 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:13.844970 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:13.844975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:13.845283 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:13.845458 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.845468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.846727 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.846731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.847658 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.847674 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.848400 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.848404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.848518 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.849269 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:13.849289 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:13.850956 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:13.852830 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.852834 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.852837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.853088 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.853368 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.853371 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.853373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.853453 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.853932 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:13.853935 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:13.854054 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 16:44:13.854148 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 16:44:13.854339 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260928 16:44:13.858775 16200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822: Bootstrap starting.
I20260928 16:44:13.861761 16200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:13.862782 16200 log.cc:824] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:13.865640 16200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822: No bootstrap required, opened a new log
I20260928 16:44:13.868988 16200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871ed195a0d44e6d83ed184615b68822" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35635 } } peers { permanent_uuid: "971f17b386f64ffe830316662ec38f0f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } } peers { permanent_uuid: "300ada7ae9854b1882898fcafa86c4ae" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } }
I20260928 16:44:13.869273 16200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 871ed195a0d44e6d83ed184615b68822, State: Initialized, Role: FOLLOWER
I20260928 16:44:13.869645 16200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871ed195a0d44e6d83ed184615b68822" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35635 } } peers { permanent_uuid: "971f17b386f64ffe830316662ec38f0f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } } peers { permanent_uuid: "300ada7ae9854b1882898fcafa86c4ae" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } }
I20260928 16:44:13.870400 16370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871ed195a0d44e6d83ed184615b68822" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35635 } } peers { permanent_uuid: "971f17b386f64ffe830316662ec38f0f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } } peers { permanent_uuid: "300ada7ae9854b1882898fcafa86c4ae" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } } }
I20260928 16:44:13.870736 16370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:13.871294 16200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:13.871408 16174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 16:44:13.871579 16174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:13.873222 16174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871ed195a0d44e6d83ed184615b68822" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35635 } } peers { permanent_uuid: "971f17b386f64ffe830316662ec38f0f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } } peers { permanent_uuid: "300ada7ae9854b1882898fcafa86c4ae" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } }
I20260928 16:44:13.876508 16324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "871ed195a0d44e6d83ed184615b68822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "300ada7ae9854b1882898fcafa86c4ae"
I20260928 16:44:13.876983 16324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 300ada7ae9854b1882898fcafa86c4ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:13.878943 16324 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 300ada7ae9854b1882898fcafa86c4ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 871ed195a0d44e6d83ed184615b68822 in term 1.
I20260928 16:44:13.879272 16275 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } attempt: 4
W20260928 16:44:13.879376 16381 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:44:13.879604 16381 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:44:13.879987 16151 leader_election.cc:302] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 300ada7ae9854b1882898fcafa86c4ae, 871ed195a0d44e6d83ed184615b68822; no voters:
I20260928 16:44:13.880373 16370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:13.880677 16370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [term 1 LEADER]: Becoming Leader. State: Replica: 871ed195a0d44e6d83ed184615b68822, State: Running, Role: LEADER
I20260928 16:44:13.877517 16132 master_runner.cc:186] Error getting master registration for 127.25.71.125:38147: OK, instance_id { permanent_uuid: "971f17b386f64ffe830316662ec38f0f" instance_seqno: 1790613853466003 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:44:13.881145 16370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871ed195a0d44e6d83ed184615b68822" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35635 } } peers { permanent_uuid: "971f17b386f64ffe830316662ec38f0f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } } peers { permanent_uuid: "300ada7ae9854b1882898fcafa86c4ae" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } }
I20260928 16:44:13.881915 16174 leader_election.cc:288] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [CANDIDATE]: Term 1 election: Requested vote from peers 971f17b386f64ffe830316662ec38f0f (127.25.71.125:38147), 300ada7ae9854b1882898fcafa86c4ae (127.25.71.124:36995)
I20260928 16:44:13.886075 16370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 871ed195a0d44e6d83ed184615b68822. Latest consensus state: current_term: 1 leader_uuid: "871ed195a0d44e6d83ed184615b68822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871ed195a0d44e6d83ed184615b68822" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35635 } } peers { permanent_uuid: "971f17b386f64ffe830316662ec38f0f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } } peers { permanent_uuid: "300ada7ae9854b1882898fcafa86c4ae" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } } }
I20260928 16:44:13.886340 16370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:13.881162 16132 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 16:44:13 dist-test-slave-nqgh krb5kdc[16119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613853, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.71.125@KRBTEST.COM for kudu/127.25.71.124@KRBTEST.COM
I20260928 16:44:13.887715 16389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:13.891625 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.880032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.71.124:36995 (local address 127.0.0.1:37896)
0928 16:44:13.880405 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.880424 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:13.880529 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:13.881382 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:13.881385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:13.881636 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:13.881773 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.881781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.883060 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.883064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.883964 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.883977 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.884770 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.884774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.884865 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.885465 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:13.885481 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:13.887228 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:13.889695 (+ 2467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.889699 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.889701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.889956 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.890459 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.890462 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.890464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.890554 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.891065 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:13.891068 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:13.891175 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 16:44:13.891269 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 16:44:13.891440 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266}
I20260928 16:44:13.892266 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.880754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37896 (local address 127.25.71.124:36995)
0928 16:44:13.880973 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:13.880978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:13.880992 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:44:13.881037 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:13.881039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:13.881108 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:13.881183 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:13.881957 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.882846 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.884166 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.884520 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.884611 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.887481 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:13.887487 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:13.887491 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:13.887550 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:13.889541 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.890245 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.890249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.890251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.890345 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.890669 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.890672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.890673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.890925 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:13.891020 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:13.891977 (+ 957us) server_negotiation.cc:299] Negotiation successful
0928 16:44:13.892112 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260928 16:44:13.893872 16389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:13.897249 16275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f: Bootstrap starting.
I20260928 16:44:13.902724 16275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:13.906165 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.887698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37908 (local address 127.25.71.124:36995)
0928 16:44:13.892460 (+ 4762us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:13.892464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:13.896752 (+ 4288us) server_negotiation.cc:407] Connection header received
0928 16:44:13.896964 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:13.896968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:13.897022 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:13.897073 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:13.898148 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.898961 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.900180 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.900523 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.900591 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.902497 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:13.902501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:13.902503 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:13.902532 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:13.904220 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.904749 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.904753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.904755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.904838 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.905136 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.905140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.905141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.905387 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:13.905487 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:13.905846 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 16:44:13.905979 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4701,"thread_start_us":149,"threads_started":1}
I20260928 16:44:13.907161 16275 log.cc:824] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:13.915719 16275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f: No bootstrap required, opened a new log
I20260928 16:44:13.916553 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.887122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42628 (local address 127.25.71.126:35635)
0928 16:44:13.887283 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:13.887287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:13.889314 (+ 2027us) server_negotiation.cc:407] Connection header received
0928 16:44:13.889700 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:13.889704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:13.889774 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:13.889839 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:13.891849 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.892932 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.907964 (+ 15032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.908368 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.908466 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.911490 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:13.911504 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:13.911507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:13.911551 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:13.913678 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.914599 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.914603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.914605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.914759 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.915262 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.915266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.915268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.915555 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:13.915673 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:13.916207 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 16:44:13.916369 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:44:13.916996 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.890048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.25.71.125:38147)
0928 16:44:13.890190 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:13.890194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:13.907363 (+ 17169us) server_negotiation.cc:407] Connection header received
0928 16:44:13.907562 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:13.907566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:13.907624 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:13.907677 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:13.909059 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.909837 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.911133 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.911442 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.911531 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.913196 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:13.913200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:13.913203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:13.913235 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:13.914832 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.915299 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.915302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.915304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.915380 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.915682 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.915685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.915687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.915917 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:13.916035 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:13.916631 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 16:44:13.916832 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:44:13.920915 16275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871ed195a0d44e6d83ed184615b68822" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35635 } } peers { permanent_uuid: "971f17b386f64ffe830316662ec38f0f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } } peers { permanent_uuid: "300ada7ae9854b1882898fcafa86c4ae" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } }
I20260928 16:44:13.921288 16275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f17b386f64ffe830316662ec38f0f, State: Initialized, Role: FOLLOWER
I20260928 16:44:13.921754 16275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871ed195a0d44e6d83ed184615b68822" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35635 } } peers { permanent_uuid: "971f17b386f64ffe830316662ec38f0f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } } peers { permanent_uuid: "300ada7ae9854b1882898fcafa86c4ae" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } }
I20260928 16:44:13.922683 16396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871ed195a0d44e6d83ed184615b68822" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35635 } } peers { permanent_uuid: "971f17b386f64ffe830316662ec38f0f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } } peers { permanent_uuid: "300ada7ae9854b1882898fcafa86c4ae" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } } }
I20260928 16:44:13.923084 16396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:13.925034 16324 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 300ada7ae9854b1882898fcafa86c4ae [term 1 FOLLOWER]: Refusing update from remote peer 871ed195a0d44e6d83ed184615b68822: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:13.926071 16383 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "300ada7ae9854b1882898fcafa86c4ae" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:13.932120 16356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 300ada7ae9854b1882898fcafa86c4ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 871ed195a0d44e6d83ed184615b68822. Latest consensus state: current_term: 1 leader_uuid: "871ed195a0d44e6d83ed184615b68822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871ed195a0d44e6d83ed184615b68822" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35635 } } peers { permanent_uuid: "971f17b386f64ffe830316662ec38f0f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } } peers { permanent_uuid: "300ada7ae9854b1882898fcafa86c4ae" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } } }
I20260928 16:44:13.932348 16356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 300ada7ae9854b1882898fcafa86c4ae [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:13.936480 16383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "871ed195a0d44e6d83ed184615b68822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871ed195a0d44e6d83ed184615b68822" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35635 } } peers { permanent_uuid: "971f17b386f64ffe830316662ec38f0f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } } peers { permanent_uuid: "300ada7ae9854b1882898fcafa86c4ae" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } } }
I20260928 16:44:13.936712 16383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:13.938292 16356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 300ada7ae9854b1882898fcafa86c4ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "871ed195a0d44e6d83ed184615b68822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871ed195a0d44e6d83ed184615b68822" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35635 } } peers { permanent_uuid: "971f17b386f64ffe830316662ec38f0f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } } peers { permanent_uuid: "300ada7ae9854b1882898fcafa86c4ae" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } } }
I20260928 16:44:13.938476 16356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 300ada7ae9854b1882898fcafa86c4ae [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:13.939016 16394 mvcc.cc:205] Tried to move back new op lower bound from 7334354345621102592 to 7334354345502167040. Current Snapshot: MvccSnapshot[applied={T|T < 7334354345621102592}]
I20260928 16:44:13.940660 16389 catalog_manager.cc:1347] Generated new cluster ID: df8c720349574c2fad60ed2e6797f7c6
I20260928 16:44:13.940755 16389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:13.942153 16397 mvcc.cc:205] Tried to move back new op lower bound from 7334354345621102592 to 7334354345502167040. Current Snapshot: MvccSnapshot[applied={T|T < 7334354345621102592}]
I20260928 16:44:13.942565 16275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:13.945436 16248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "871ed195a0d44e6d83ed184615b68822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "971f17b386f64ffe830316662ec38f0f"
I20260928 16:44:13.945976 16249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:13.950311 16249 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f [term 1 FOLLOWER]: Refusing update from remote peer 871ed195a0d44e6d83ed184615b68822: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260928 16:44:13.951658 16411 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:44:13.951778 16411 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:44:13.951977 16248 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 871ed195a0d44e6d83ed184615b68822 in term 1.
I20260928 16:44:13.959304 16370 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "971f17b386f64ffe830316662ec38f0f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 16:44:13.962039 16228 tablet.cc:2369] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:44:13.964998 16396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 871ed195a0d44e6d83ed184615b68822. Latest consensus state: current_term: 1 leader_uuid: "871ed195a0d44e6d83ed184615b68822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871ed195a0d44e6d83ed184615b68822" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35635 } } peers { permanent_uuid: "971f17b386f64ffe830316662ec38f0f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } } peers { permanent_uuid: "300ada7ae9854b1882898fcafa86c4ae" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } } }
I20260928 16:44:13.965184 16396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:13.965293 16396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "871ed195a0d44e6d83ed184615b68822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871ed195a0d44e6d83ed184615b68822" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35635 } } peers { permanent_uuid: "971f17b386f64ffe830316662ec38f0f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } } peers { permanent_uuid: "300ada7ae9854b1882898fcafa86c4ae" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } } }
I20260928 16:44:13.965373 16396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:13.973311 16413 mvcc.cc:205] Tried to move back new op lower bound from 7334354345621102592 to 7334354345502167040. Current Snapshot: MvccSnapshot[applied={T|T < 7334354345621102592}]
I20260928 16:44:13.977451 16383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "871ed195a0d44e6d83ed184615b68822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871ed195a0d44e6d83ed184615b68822" member_type: VOTER last_known_addr { host: "127.25.71.126" port: 35635 } } peers { permanent_uuid: "971f17b386f64ffe830316662ec38f0f" member_type: VOTER last_known_addr { host: "127.25.71.125" port: 38147 } } peers { permanent_uuid: "300ada7ae9854b1882898fcafa86c4ae" member_type: VOTER last_known_addr { host: "127.25.71.124" port: 36995 } } }
I20260928 16:44:13.977622 16383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:13.983989 16389 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:13.985523 16389 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:14.002821 16389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 871ed195a0d44e6d83ed184615b68822: Generated new TSK 0
I20260928 16:44:14.004045 16389 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:44:14.034178 16353 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:14.032087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37910 (local address 127.25.71.124:36995)
0928 16:44:14.032230 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:14.032235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:14.032979 (+ 744us) server_negotiation.cc:407] Connection header received
0928 16:44:14.033195 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:14.033199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:14.033251 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:14.033302 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:14.034029 (+ 727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37910: BlockingRecv error: recv got EOF from 127.0.0.1:37910 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 16:44:14.036933 16203 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:14.035130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42632 (local address 127.25.71.126:35635)
0928 16:44:14.035253 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:14.035257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:14.035962 (+ 705us) server_negotiation.cc:407] Connection header received
0928 16:44:14.036132 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:14.036135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:14.036186 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:14.036235 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:14.036779 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42632: BlockingRecv error: recv got EOF from 127.0.0.1:42632 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 16:44:14.038754 16280 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:14.035692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.25.71.125:38147)
0928 16:44:14.035806 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:14.035810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:14.037732 (+ 1922us) server_negotiation.cc:407] Connection header received
0928 16:44:14.037905 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:14.037908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:14.037964 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:14.038014 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:14.038562 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43518: BlockingRecv error: recv got EOF from 127.0.0.1:43518 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 16:44:14.044226 16280 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:14.042063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43526 (local address 127.25.71.125:38147)
0928 16:44:14.042183 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:14.042187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:14.042950 (+ 763us) server_negotiation.cc:407] Connection header received
0928 16:44:14.043209 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:14.043213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:14.043303 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:14.043369 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:14.044098 (+ 729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43526: BlockingRecv error: recv got EOF from 127.0.0.1:43526 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 16:44:14.049752 16280 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:14.048774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43542 (local address 127.25.71.125:38147)
0928 16:44:14.048922 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:14.048926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:14.048940 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:44:14.049000 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:14.049003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:14.049050 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:14.049135 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:14.049624 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43542: BlockingRecv error: recv got EOF from 127.0.0.1:43542 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 16:44:14.055171 16280 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:14.054276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43552 (local address 127.25.71.125:38147)
0928 16:44:14.054401 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:14.054405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:14.054418 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:44:14.054472 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:14.054474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:14.054517 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:14.054583 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:14.055065 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43552: BlockingRecv error: recv got EOF from 127.0.0.1:43552 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 16:44:14.062953 16280 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:14.061718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.25.71.125:38147)
0928 16:44:14.061857 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:14.061860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:14.062089 (+ 229us) server_negotiation.cc:407] Connection header received
0928 16:44:14.062220 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:14.062223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:14.062276 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:14.062350 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:14.062820 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43560: BlockingRecv error: recv got EOF from 127.0.0.1:43560 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 16:44:14.074644 16280 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:14.073339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43568 (local address 127.25.71.125:38147)
0928 16:44:14.073566 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:14.073571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:14.073769 (+ 198us) server_negotiation.cc:407] Connection header received
0928 16:44:14.073901 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:14.073904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:14.073958 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:14.074024 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:14.074494 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43568: BlockingRecv error: recv got EOF from 127.0.0.1:43568 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
W20260928 16:44:14.094182 16280 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:14.092923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43574 (local address 127.25.71.125:38147)
0928 16:44:14.093084 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:14.093115 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:14.093350 (+ 235us) server_negotiation.cc:407] Connection header received
0928 16:44:14.093488 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:14.093490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:14.093538 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:14.093605 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:14.094055 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43574: BlockingRecv error: recv got EOF from 127.0.0.1:43574 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 16:44:14.129657 16280 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:14.128481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43578 (local address 127.25.71.125:38147)
0928 16:44:14.128620 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:14.128625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:14.128763 (+ 138us) server_negotiation.cc:407] Connection header received
0928 16:44:14.128911 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:14.128914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:14.128967 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:14.129035 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:14.129508 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43578: BlockingRecv error: recv got EOF from 127.0.0.1:43578 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 16:44:14.197685 16280 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:14.196246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43592 (local address 127.25.71.125:38147)
0928 16:44:14.196427 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:14.196432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:14.196449 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:44:14.196577 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:14.196580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:14.196634 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:14.196701 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:14.197508 (+ 807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43592: BlockingRecv error: recv got EOF from 127.0.0.1:43592 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260928 16:44:14.329599 16280 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:14.328230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43606 (local address 127.25.71.125:38147)
0928 16:44:14.328395 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:14.328399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:14.328506 (+ 107us) server_negotiation.cc:407] Connection header received
0928 16:44:14.328669 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:14.328672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:14.328735 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:14.328869 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:14.329460 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43606: BlockingRecv error: recv got EOF from 127.0.0.1:43606 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 16:44:14.590029 16280 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:14.588378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.25.71.125:38147)
0928 16:44:14.588574 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:14.588579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:14.588648 (+ 69us) server_negotiation.cc:407] Connection header received
0928 16:44:14.588931 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:14.588934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:14.589007 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:14.589143 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:14.589893 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42180: BlockingRecv error: recv got EOF from 127.0.0.1:42180 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 16:44:14.832643 16367 catalog_manager.cc:1259] Loaded cluster ID: df8c720349574c2fad60ed2e6797f7c6
I20260928 16:44:14.832767 16367 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 300ada7ae9854b1882898fcafa86c4ae: loading cluster ID for follower catalog manager: success
I20260928 16:44:14.834961 16367 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 300ada7ae9854b1882898fcafa86c4ae: acquiring CA information for follower catalog manager: success
I20260928 16:44:14.836676 16367 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 300ada7ae9854b1882898fcafa86c4ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:44:14.953606 16411 catalog_manager.cc:1259] Loaded cluster ID: df8c720349574c2fad60ed2e6797f7c6
I20260928 16:44:14.953732 16411 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f: loading cluster ID for follower catalog manager: success
I20260928 16:44:14.955897 16411 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f: acquiring CA information for follower catalog manager: success
I20260928 16:44:14.957450 16411 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 971f17b386f64ffe830316662ec38f0f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:44:15.110100 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:15.105136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42192 (local address 127.25.71.125:38147)
0928 16:44:15.105567 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:15.105571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:15.105591 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:44:15.105681 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:15.105684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:15.105845 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:15.105975 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:15.106368 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.106994 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.108082 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.108335 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.108436 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:15.109017 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:15.109543 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:15.109656 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:15.109816 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 16:44:15.109887 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":140,"threads_started":1}
I20260928 16:44:15.119590 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:15.115022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.25.71.124:36995)
0928 16:44:15.115352 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:15.115357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:15.115472 (+ 115us) server_negotiation.cc:407] Connection header received
0928 16:44:15.115699 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:15.115702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:15.115836 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:15.115943 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:15.116262 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.116822 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.117778 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.118007 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.118095 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:15.118541 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:15.119119 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:15.119254 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:15.119332 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 16:44:15.119404 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":86,"threads_started":1}
I20260928 16:44:15.128079 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:15.123886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.25.71.124:36995)
0928 16:44:15.124023 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:15.124028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:15.124226 (+ 198us) server_negotiation.cc:407] Connection header received
0928 16:44:15.124403 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:15.124407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:15.124472 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:15.124517 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:15.124931 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.125654 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.126752 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.127119 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.127249 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:15.127538 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:15.127627 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:15.127693 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:15.127831 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 16:44:15.127908 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:44:15.129015 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:15.124360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.25.71.125:38147)
0928 16:44:15.124521 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:15.124525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:15.124660 (+ 135us) server_negotiation.cc:407] Connection header received
0928 16:44:15.124914 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:15.124919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:15.124989 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:15.125077 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:15.125422 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.126317 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.127691 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.128054 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.128166 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:15.128540 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:15.128621 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:15.128678 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:15.128808 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 16:44:15.128883 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:44:15.129352 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:15.123783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50346 (local address 127.25.71.126:35635)
0928 16:44:15.124085 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:15.124090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:15.124371 (+ 281us) server_negotiation.cc:407] Connection header received
0928 16:44:15.124498 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:15.124501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:15.124693 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:15.124847 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:15.125284 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.126332 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.127379 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.127785 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.127884 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:15.128273 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:15.128790 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:15.128884 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:15.128957 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 16:44:15.129039 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":89,"threads_started":1}
I20260928 16:44:15.140273 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 16132
I20260928 16:44:15.148753 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 16207
I20260928 16:44:15.157140 25885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskp76gbp/build/debug/bin/kudu with pid 16281
2026-09-28T16:44:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3291 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6506 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (309478 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST